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ABSTRACT 


The  thesis  develops  principles  for  designing  machines  to  support 
vircual  computer  systems.  The  virtual  computer  system  (VCS)  is  an 
important  construct  which  arises  as  a  solution  to  a  problem  of 
present  computer  system  technology.  The  most  important  new  result 
of  the  thesis  is  the  model  of  a  process  running  on  a  virtual  computer 
system  and  the  derivation  of  design  principles  from  that  model.  The 
approach  adopted  is  to  consider  the  introduction  of  VCS's  into  the 
rich,  complex  architectures  likely  to  be  found  in  IV  generation 
systems.  Because  of  the  additional  system  structure  in  the  IV  genera¬ 
tion,  the  somewhat  ad  hoc  third  generation  virtual  machine  software 
mapping  techniques  should  be  doomed  to  failure.  This  result  leads 
us  away  from  an  interpretation  of  virtual  machines  that  depends 
implicitly  on  techniques  used  in  third  generation  systems.  Instead, 
we  are  able  to  develop  a  generalized  model  of  a  process  running  on  a 
IV  generation  VCS.  The  model  allows  us  to  understand  different 
properties  of  virtual  machines  and  to  interpret  a  number  of  proposed 
implementations  of  VCS's  in  terms  of  the  model.  Furthermore,  the 
model  leads  naturally  to  an  implementation  of  virtual  machines,  the 
Hardware  Virtualizer  (HV) ,  which  provides  an  efficient  and  simplified 
mechanism  for  virtual  machines.  A  number  of  detailed  examples  illus¬ 
trate  how  *-he  Hardware  Virtualizer  might  operate  in  an  actual  IV 
generation  system. 
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SYNOPSIS 


The  thesis  develops  principles  for  designing  machines  to 
jupcort  virtual  computer  systems.  The  virtual  comcuter  system 
(VC S)  is  an  important  construct  which  arises  as  a  solution  tc  a 
particularly  vexing  oroDleir  cf  oresent  computer  system 
techno  I ogy . 

While  the  recent  development  of  large  mu  1 1 i -access* 
Hu  I  ti-orogramming ,  multi-processing  systems  has  simplified  and 
improved  access  to  computer  systems  by  the  bjl*«  of  the  user 
community,  there  has  been  one  important  class  of  users  unc-rle  to 
r'ofit  from  these  recent  advances.  This  class  is  tne  system 
programmers  whose  programs  must  be  run  on  a  I23C5  machine  ard  not 
on  an  extended  machine,  e.g.  under  the  operating  system.  System 
programs,  e.g.  other  operating  systems  or  different  versions  of 
the  S3me  operating  systen,  require  direct  addressability  to  the 
resources  of  the  system  and  do  not  call  uoor.  the  operating  sys  +  em 
to  manage  these  resources.  Thus,  system  programs  may  not 
co-exist  with  normal  production  uses  of  tne  system.  This 
situation  has  forced  most  installations  into  clumsy  and 
inefficient  administrative  solutions  such  as  "stand-s I  one" 
debugging. 

Recently,  a  techrioue  for  resolving  tnese  difficulties  has 
been  devised.  The  solution  utilizes  a  construct  called  a  virtual 
comcuter  system  or  virtual  machine  which  is,  b3sicslly,  a  very 
efficient  simulated  cooy  (or  copies)  of  the  bare  host  mac.nine. 
These  copies  differ  from  each  other  and  from  tne  host  only  in 
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Their  exact  con f  igurst ions,  e.g.  the  amount  of  memory  or 
particular  I/C  devices  attached.  Therefore#  not  only  stanoard 
jser  programs  but  system  programs  and  complete  operating  systems 
that  run  on  the  real  computer  system  will  run  on  the  VCS  with 
identical  results  [except  for  certain  special  timing 
le&ender.ci  es)  .  Thus,  me  VCS  orcvides  a  generalization  ever 
familiar  systems  by  also  being  a  mu  1 1 i -env ironmen t  system. 


The  recent  origin  of  the  field  nas  forced  the  development  of 
i  terminology  t0  represent  ideas  in  VCS*s.  Chapter  2  introduces 
some  basic  notions  in  VCS’s  and  proposes  terminology  to  represent 
them.  The  terminology  is  new  but  some  excerots  were  oreviously 
published  by  the  author  in  "Virtual  Machires?  Semantics  and 
Examples"  1521. 

Despite  the  rather  significant  benefits  that  derive  from 
virtual  machines.  only  a  limited  number  of  current  systems 
feature  this  facility.  This  is  largely  due  to  the  unsuitable 
nature  of  existing  hardware.  In  order  to  support  a  VCS  on 
contemporary  eauipment.  a  particular  software  construction  must 
be  employed.  However,  this  construction  ma.y  only  be  aoDlied  to 
machines  with  certain  properties.  Chapter  3  examines  the 
construction  of  VCS’s  on  third  generation  computer  systems  and 

derives  a  set  of  empirical  retirements  to  deterrine  if  a  machine 
may  be  virtualized.  The  appendices  treat  the  application  of 
these  empirical  rules  to  a  number  of  case  studies.  This  material 
is  new  although  some  preliminary  results  were  reported  by  the 
author  in  "Virtual  Machine  Systems"  1541  and  "Hardware 
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The  remainder  of  the  thesis  (Chaoter  4)  develops  a  structure 
for  computer  systems  that  permits  the  orderly  introduction  of 
VCS*s  into  future  computer  systeirs,  IV  generation  computer 
systems  will  likely  feature  a  formal  implementation  (in  firmware) 
of  the  process  model.  Because  of  the  existence  of  a  large 
database  needed  by  the  firmware  (and  or ivi I eseo  software)  to 
support  the  process  model*  it  will  be  difficult  to  employ  the 
software  mapping  technicues  used  in  III  generation  >/ CS*s.  A 
model  is  developed  to  represent  the  execution  of  processes  on  a 
IV  generation  VCS.  The  model  features  two  mapst  (1)  a  process 
map  called  *  which  maps  process  names*  e.g.  segments*  semaphores* 
orocess-id* s»  into  resource  names*  e.g.  memory  locaticrs* 
processor  numoers*  and  (H)  a  virtual  machine  mac  (Vkmap)  f  which 
maps  virtual  resource  names  into  real  resource  names.  The 
process  map  is  strictly  an  intra-level  mac  expressing  a 
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relationship  within  a  virtual  machine*  the  VHmao  is  an 
inter-level  map  expressing  a  relationship  tetween  (the  resources 
of)  two  adjacent  levels  of  (virtual)  machines.  Thus*  the  action 


of  running  a  process  on  a  VCS  consists  of  running  it  under  the 
composed  map  f  o  ft.  If  the  VCS  itself  is  running  a  VCS  under  it, 
then  the  action  consists  of  running  a  process  under  f  o  f  o  6. 
Thus*  for  VCS  recursion,  only  the  f  mao  must  be  invoked 
recursively*  not  the  process  map  G.  This  result  implies  that  for 
a  complex  G  but  a  rudimentary  f,  recursion  snoui-d  be  easy. 

The  TV  generation  VCS  model  leads  directly  to  a  design*  the 
Hardware  Virtualizer  (HV) *  for  proposed  implementations  of  IV 
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generation  V CS*s.  The  aesign  has  the  characteristic  that  all 
arocess  exceptions  are  hincled  directly  within  the  executing  VCS 
without  software  intervention.  All  resource  faults  (VM-fsults) 
by  a  VCS  are  directed  to  its  Virtual  Machine  Monitor  (VMM; 
without  Knowledge  of  processes  on  the  VCS.  Fault  handling- 
invocation  and  execution  of  VCS*s  works  directly  regardless  of 
r ecursi on. 

Detailed  illustrations  are  provided  for  several  possible 
choice^  of  the  VMmap  f,  e.g.  relocation  and  bounds  (R-8) *  paging. 
Preliminary  performance  estimates  indicate  that  for  an  R-B  VMirap 
f  (with  associative  iremory),  VCS  performance  will  be 
approximately  that  of  the  real  machine,  regardless  of  the  death 
of  recursion.  A  paged  VMmap  f  provides  performance  comoarafcle  to 
the  real  machine  for  several  levels  of  recursion  over  a  range  of 
likely  onerating  conditions. 

A I  though  the  model  and  propcsec  implementations  cf  Chapter  4 
arise  in  an  attempt  to  guarantee  virtualization  for  IV  generation 
architectures,  by  suitable  simplification  and  intercretat cn  of 
the  model.  the  orincioles  which  emerge  ere  aoolicable  to  ether 
architectures,  as  well.  In  particular,  the  model  suggests 
introducing  paging  in  the  IPM  36C/67  as  a  formal  f-mao.  rather 
than  an  aa  hoc  4-mao  as  was  done.  This  leans  to  a  greatly 
simplified  structure  for  CP-67. 

All  of  the  material  presented  in  Chapter  4,  the  IV 
generation  VCS  model,  suggested  implementations,  performance 
expectations,  examples,  etc.,  is  new  and  original.  An  early 
oroposal  for  a  f irmware-assisted  implementation  of  v OS’s  was 
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CHAPTFR  1. 


INTRODUCTION 


1.0  OVERVIEW 
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The  Need  lor  Virtual  bailees 


Recent  dev?  I  oprer.  ts  in  ccm-outer  system  design  have  been 
directed  toward  im proving  the  ease  of  com outer  use  by  the  normal 
user  peculation.  Seme  cf  the  more  Dramatic  dev-3 1  ODirents  irclude 
the  introduction  cf  l3rge  mu  1 1 i- see  ess »  mu  I  t  i-cr egramm inc,  and 
multi- crocessirg  systems.  These  systems  have  eliminated  the  need 
for  physical  access  to  tne  train  computing  facility  by  the  users, 
and  for  direct  software  access  to  the  system  resources  by  the 
users*  programs.  Thus,  users  sit  at  teletypes  cr  submit  Jots  via 
remote  card  readers  and  need  not  be  concerned  either  with  the 
status  of  lights  or  switches  on  the  central  crocesser,  or  with 
the  allocation  of  memory,  I/O  devices,  or  processor  time  to  their 
oregrams.  These  functions  are  now  performed  ( transparent  ly  to 
the  users)  by  the  operating  system.  When  either  users  or  their 
orogrsms  reouiry  services  to  be  performed,  they  call  upor  the - 
ooeratirq  system  which  performs  the  actual  m3niou»5Ticn  and 
allocation  cf  re- ounces.  In  some  sense,  the  operating  system 
{together  with  the  nardwsre)  provides  an  idealized  or  extended 
view  of  a  computer  sytem.  Thus,  program  preparation}  debugging, 
and  running  has  Become  significantly  easier  for  most  of  the  user 
ooou I  at  ion. 
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Unfortunately,  one  yery  important  group  of  users  has  been 
jnst.lp  to  rs?D  the  benefits  of  these  cevelopments  and  advances. 
These  ~.r  >  the  users  who*  for  3  variety  of  reasons*  desire  or  need 
Jirecf  access  to  the  resources  of  the  system.  Tneir  programs  are 
written  to  run  on  a  5sce  machine.  Calling  ucon  an  operating 
system  fthe  so-called  extended  UUCiDiDS*  will  not  oo  for  then.  We 
will  term  such  users  Sj'l.ten;  programmers  and  the  programs  they 
les'.re  to  run,  system  programs.  Because  system  programs  canrot 
''un  under  t  he  normal  operating  system  in  the.  oroduction 
environment,  it  is  usually  recessary  to  treat  trem  as  special 
casfs.  When  they  must  be  run,  the  normal  ooerating  system  is 
orougnt  dcwr  and  a  system  program  may  run  "stand-alone."  This 
practice  xs  inconvenient  for  both  the  system  programmers  and  the 
normal  users,  and  inefficient  srd  wasteful  cf  the  system 
re.curccs.  Turrhermore,  it  can  introduce  edditionl  personnel  and 
procedural  complexities.  Indeed,  it  is  ironic  tnst  those  who 
develop  and  maintain  operating  systems  are  often  the  very  people 
who  Tjke  the  least  advantage  of  their  efforts. 


a  few  specific  examples  of  some  system  programming  tasks 
might  clarify  these  coirts. 

(1>  Debugging  the  operating  system--  The  operating 
system  is  written  tc  run  on  the  bare  machire. 
Currently,  deputing  and  improving  the  operating  system 
is  treated  as  a  second  class  activity,  often  assigned 
*o  computer  time  in  the  middle  of  i he  night. 

(?)  Running  diagnostic  software--  The  software  for 
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testing  malfunctions  in  the  various  units  ir  the 
system,  e.g.,  is  a  face  drive  recordirq  correctly,  is  a 
disk  drive  seeking  correctly,  etc.  normally  runs 
stand-alone  or  the  bare  machire  or  cn  a  rud i mentary 
monitor  (Test  anc  Diagnostic  Monitor)  which  itself  runs 
on  a  bare  machine. 

(3)  punning  :ther  operating  systems--  Vihen  there  is 
mere  than  ere  operating  system  for  a  given  computer 
system  (or  even  different  releases  of  the  same 
cceratinq  system)  it  is  often  easier  to  run  a  subsystem 
(compiler,  interpreter,  etc.)  under  its  cwr  operating 
system  than  to  take  the  time  and  cost  to  convert  it  to 
another  ocerating  system. 


iiBsi  ics  viciyal  fjaiSiossl 

Pecentiy,  a  technicue  for  resolving  these  difficulties  has 
oeer  deviser.  The  solution  utilizes  a  construct  called  a  Jiic.ty.3i 
£S££yiIE  system,  (VCS)  or  Jiiclysi  IsasfciEi  •  t  Section  2.1  covers 
•most  of  this  termi  nc  I  egy .  1  Basically  a  virtual  machine  is  a  very 
efficiert  simulatec  ccoy  (or  copies)  of  the  bare  host  machire. 
Because  of  its  functionality,  it  is  possible  tc  run  system 
programs J  because  cf  its  efficiency,  it  is  reasonable  to  run 
them  in  the  normal  production  environment.  The  program  which 
mediates  between  the  virtual  machire  ard  the  actual  resources  of 
the  system  (tne  real  cr  host  machine)  is  called  the  Jiiciyfli 
ISCtiD®  jpgn imor  (VMh)  .  Since  the  virtual  machine  is  identical  to 
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thf  host  r.pchino  ir  cl  I  sigrificant  respects*  the  reed  rot 

amrioy  thr  faniiia<~  instruct icn-by-ins troct i cn  software 

i  r  tcpor  et  ?t  ion  techniou?  that  normal  Iv  ^ust  be  used*  Esther* 

virtual  machines  nay  be  constructed  ir  such  a  way  that  most 

instructors  of  the  vir*'j-»l  machine  execute  on  the  host  cirectly. 

Just  those  instructions  which  cannot  he  permitted  tc  execute 

lirectly  must  be  interpreter.  Thus*  we  ircorporste  these  rcticns 

into  our  definition  of  a  virtual  computer  system  ir  orcer  to 

iistir.guish  it  from  a  number  of  ether  objects  which  have  often 

oeer  casually  callec  virtual  "machines. 

A  virtual  ccmouter  system  is  a  her owsre-so f tware 
duplicate  of  a  real  existing  computer  system  in  which  3 
statistically  dominant  subset  of  the  virtual 
processor’s  instructions  execute  on  the  host  processor 
ir  native  mode. 

Thus*  a  VCS  proviles  an  efficient  oreraticr  of  ore  or  mere 
cocies  cf  a  comolefe  computer  system,  similar  to  the  host  (or 
~e;  f)  system.*  These  cooics  differ  from  each  other  anc  frem  the 
host  or  |y  ir  their  exact  configurations,  e.g.  the  amourt  of 
Tierrcrv  or  the  particular  I/O  device,  attached.  The  virtual  and 
real  processors  must  either  be  identical  or  members  of  the  same 
computer  family,  e.c.  ICh  ^vsteir/?6G-?7C.  Therefore,  not  orly 
stardarc  user  programs  tut  system  programs  anc  complete  operating 
systems  that  run  on  the  reel  computer  system  will  run  or  the  VCS 
with  identical  results  lexcent  for  certair  special  timing 
jependenc i esl .  Thus  the  VCS  previces  a  gener a  I  i as t i cn  over  the 
familiar  mu  I  ti -access  *  mu  1 1  i -programming  ,  mu  It  i-crocessinc* 
systems,  by  also  providing  a  multi-envi roniren t  system. 

Tr  addition  tc  oermittirc  the  three  system  programming 
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examples,  above,  to  coexist  with  normal  orcduction  uses  of  the 
system*  virtual  machines  introduce  sn  additional  degree  of  system 
flexibility.  This  flexioility  is  a  consequence  of  the  additional 
level  of  binding  in  a  virtual  machine,  between  the  rescurc 
referenced  by  system  programs  running  on  the  VCS  and  the  actual 
resources  that  they  ccrresDond  with.  Some  immediate  advantages 
are  I 

(1)  Punning  with  3  virtual  configuration  which  is 
different  from  the  real  configuration--  This  use  can  be 
important  for  running  systems  with  more  virtual  memory 
or  processors  than  actually  exist,  or  debugging 
computer  retwcrks  and  telecommunications  applications. 

(2)  Measuring  operating  systems--  Since  the  VMM 
mediates  between  the  virtual  and  real  resources  of  a 
system,  it  can  measure  how  an  operating  system  on  a 
virtual  machira  is  manipulating  its  resources  (without 
requiring  a  modification  to  the  operating  system). 

(3)  Adding  hardware  enhancements  to  a  system--  If 
hardware  enhancements,  e.g.  paging,  are  added  tc  the 
host  machine,  it  may  still  be  possible  to  run  o  virtual 
machine  which  does  not  incorporate  these  enhancements. 
Under  these  conditions  the  operating  system  runnirg  on 
the  virtual  machine  neec  not  be  modified.  Thus,  with 
relatively  simple  VMM  modifications,  the  hardware 
enhancements  can  be  utilized  to  orovide  improved 
resource  handling  and  capabilities. 
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Syccsai  virtual  ijactiDS 

Qesoite  the  rather  si  cr.i  f  icant  beneMts  th^t  derive  from 
virtual  machines*  only  a  limitec  number  cf  current  systems 
feature  this  facility.  This  situation  is  a  result  cf  (i)  the 
-ec -nr  orioin  o*  the  ccr.ceots,  (25  misunderstandings  of  the 
technioues  for  imo I ementi n?  virtual  machines*  and  (3)  the  largely 
unsuitable  nature  of  existing  hardware.  Thus*  since  existing 
computer  systems  were  not  designee  to  sucoort  virtual  machines* 
trying  to  implement  then-  is  largeiy  a  hit-oi — miss  nrccositicn. 

Tn  the  thesis,  we  exolore  the  orohleros  associated  with 
i  me  I  ement  i  ng  virtual  machines  or  c  o>  t  errpo  r  ary  hardware.  By 
developing  an  emoirical  oasis  and  a  set  of  hardware  "ules  for 
determining  which  existirg  systems  sucoort  virtual  machines*  we 
are  able  to  verify  the  inaceauocy  of  current  designs  [Chapter  31. 

fit  last,  there  is  a  growing  realization  of  tKe  imoortarce  of 
designing  machines  which  are  v i r tua  I  i zab I e .  i.e.  suoccrt  virtual 
machines.  To  date,  there  seem  to  be  two  ocoosing  schools  of 
thought.  While  both  views  have  some  validity,  both  have 
significant  disadvantages. 

The  first  view  [5?]»  tyDifiea  by  Lauer  and  Snow  [73],  argues 
that  since  the  virtual  machine  murt  have  all  the  functionality  of 
m  real  machine,  the  simpler  the  real  machine  is  the  easier  will 
oe  the  virtual  machine  construction.  tSee  Secticr  ?.4  and  4.8.J 
In  particular,  thay  suggest  eliminating  supervisor  state*  memcry 
•napping,  etc.  Hnat  they  introduce  instead  is  a  special 
relocation-bounds  type  memory  mao,  in  which  the  aosolutg  cortents 
of  the  register  may  be  added  to  but  not  read  or  written.  Such  an 
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approach  greatly  simplifies  virtual  machire  construction,  but  it 
nas  as  a  major  weakness  that  the  virtual  machine’s  programming 
environment  is  rather  severe  and  devoid  of  structure.  This  fact 
implies  that  the  development  of  large,  interesting  systems  on  the 
virtual  machine  ma  be  cifficult.  The  authors  observe  that  their 
orocosal  "...  lacks  some  of  the  important  features  of  modern 
systems,  particularly  segmented  virtual  memories  and  a  suitable 
parameter  passing  mechanism. • 1761. 

The  second  view,  introduced  by  U.O.  Gagl iardi  and  the  author 
1511  argues  that  in  complex  (likely  XV  generation)  computer 
systems  with  a  firmware  implementation  of  the  process  model  and 
layered  segmented  address  structure,  there  will  be  a  need  for 
firmware  support  of  virtual  machines  as  well.  (See  Section  2.4 
and  4.4.1  Wh3t  emerges  is  a  proposal  which  directly  supports  the 
image  of  a  process  executirg  on  a  virtual  machine.  The  major 
advantage  of  th*s  proposal  is  that  it  is  aocticable  to  very 
complex  computing  environments  that  are  likely  to  be  developed  in 
the  near  future.  The  principal  disadvantage  is  that,  since  this 
aoproach  does  not  propose  a  direct  hardware  resource  map,  a 
certain  amount  of  software  intervention  is  still  reauired.  In 
addition,  there  are  some  fundamental  limitations  on  the  kincs  of 
recursive  structyres  (running  a  VMM  on  a  virtual  machine)  that 
can  be  supported. 
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•nocel  for  running  a  crocess  on  a  complex  (IV  generation)  computer 
system  tSec"ion  4.23.  The  model  identifies  a  formal  resource  map 
(virtual  machine  mac)  and  distinguishes  it  from  the  more  familiar 
orocess  mac»  This  allows  us  to  understand  a  number  of  very 
coir.clex  ohenomena  ir  a  relatively  simple  way*  and  permits  us  to 
interpret  the  two  earlier  proposals  [51,763  as  merely  scecial 
(sub-optimal)  applications  of  the  model  [Sections  4.4  and  4.83. 
Furthermore,  the  model  leads  us  directly  to  an  optimal  proposed 
implementation  [Section  4.53  which  is.  in  some  sense,  a  synthesis 
of  the  better  ideas  of  the  two  previous  proposals.  Thus,  the 
proposal  is  explicable  to  the  complete  range  of  computer  systems, 
includino  the  comolex  (TV  generation)  future  systems,  yet  retains 
the  directress  ano  simplicity  of  a  hardware  resource  mao  which 
needs  little  software  sucoort.  In  addition,  since  this 
i mo  lemertat ion  is  derived  directly  from  the  mocel,  the  recursive 
invocation  of  virtual  machines,  i.e.,  running  a  V*M  under  a  VMM 

ate.,  ooses  no  additional  problems.  Furthermore.  the 
implementation  should  be  very  efficient.  While  performance 
measures  are  hard  to  compare,  there  is  evidence  that  for  certain 
lihely  choices  of  maos,  the  performance  of  the  virtual  machine 
will  he  extremely  close  to  that  of  the  real  machine.  Although 
the  mocel  and  orooosed  implementations  of  Chapter  4  arise  in  an 
attempt  to  guarantee  virtualization  for  IV  generation 
achi tec tures,  by  suitable  simplification  and  interpretation  of 
the  model  the  principles  which  emerge  are  acolicable  to  other 


generations  of  architectures  (Section  4.83. 


machine  will  be  vir  tua  I  ise?t<  I  e.  Thus*  the  di  sss-rinat  ion  of  these 
principles  should  have  3  highiy  sicnificant  imssct  uson  both  the 
theoretical  and  oractical  aspects  of  corrouter  science. 


Coirputer  architecture*  like  other  arc*it 
art  of  de termini ng  the  needs  of  the  user 
and  then  cesigr.inq  to  meet  those  needs 
as  possible  within  economic  ar.1 
constraints.  Architecture  irust  inclu 
considerations*  sc  that  the  design  will 
and  feasible:  but  the  emchasis  in  arcni 
the  the  needs  of  the  user*  whereas  in 
emphasis  is  or  the  needs  of  the  fabricat 


ecture*  is  the 
cf  ?  structure 
as  effectively 
techro I cgical 
de  engineering 
be  economical 
tecture  is  upon 
erqineerirg  the 
or  121]. 
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1.1  PLAN  CF  THE  thesis 

The  remainder  of  the  thesis  is  divided  into  four  chapters* 
Each  chapter  is  preceded  by  an  introcuction  which  outlines  the 
naterial  to  follow*  but  perhaps  some  brief  organizations!  remarKs 
nere  will  be  helpful* 

The  most  important  new  result  of  the  thesis*  the  model  of  a 
orocess  running  on  a  virtual  computer  system  (VCS)  and  the 
lerivation  of  design  principles  from  that  model*  is  presented  in 
Chapter  4.  This  chapter  is  largely  self-contained*  and  the 
reader  know  ledgeable  about  current  virtual  machine  applications 
and  technology  should  be  able  to  skip  directly  to  Cnaoter  4*  if 
lesired.  The  approach  adopted  in  this  chapter  is  to  consider  the 
introduction  of  VCS's  into  the  rich*  complex  architectures  likely 
to  be  found  in  IV  generation  systems.  Peceuse  of  the  additional 
system  structure  in  the  IV  generation*  the  somewhat  ad  hoc  third 
generation  virtual  machine  software  mapping  tecbniaues  should  be 
doomed  to  failure  [Sections  4.1  and  4.33.  Thi'-,  result  leads  us 
away  from  an  interpretation  of  virtual  machines  that  depends 
•molicitly  cn  techniques  used  ir  third  generation  systems* 
Instead*  we  are  able  to  develop  a  generalized  model  of  3  process 
running  on  3  IV  generation  VCS  [Section  4.23.  The  model  allows 
us  to  understand  different  properties  of  virtual  machines  ard  to 
interpret  a  number  of  proposed  imo  lemenf ations  of  VCS*s  in  terms 
of  the  model  [Section  4.41.  Furthermore*  the  model  leads 
naturally  to  an  i mpl emen tat i on  of  victual  machines*  the  Hardware 
Virtualizer  (HV)  which  provides  an  efficient  ard  simplified 
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mechanism  for  virtual  machines  [Section  4.5).  &  number  of 
Jetailea  examples  illustrate  how  the  Hardware  Virtualizer  might 
operate  in  an  actual  IV  generation  system  [Section  4.6).  In 
addition,  the  principles  developed  in  this  chapter  are  generally 
applicable  to  other  architectures  as  well  as  the  highly  complex 
IV  generation.  Proper  simplification  and  interpretation  of  the 
model  leads  to  r etrosoecti ve  principles  for  how  the  second  ano 
third  aeneration  systems  SbSiilSl  have  been  construct'd  to  support 
virtual  machines  ISection  4.8). 

The  earlier  chapters  a  basic  U'«de?  s-'-socirq  of 
virtual  machines  ana  existing  techr.iaues  for  i  me  *  e-rent  inq  them, 
and  provide  additional  motivation  for  seeking  tt-e  brfOkthroughs 
lescribec  in  Chaoter  4.  In  Chapter  2  we  introduce  seme  of  the 
oasic  virtual  machine  terminology  and  properties,  and  survey 
existing  implementations  and  related  literature.  The  aoproach  we 
take  to  terminology  in  Chapter  2  is  largely  descriotive  and 
qualitative.  Later,  in  Section  4.7,  a  number  of  those  notions 
ere  re-cast  in  terms  of  the  VCS  model  of  Chapter  '« * 

Chaoter  3  examines  the  existing  *ecb;.c  I  ogy  for  third 
generation  virtual  machines  and  shc><s  why  ^nd  how  it  is 
inacoquate.  From  the  construction  used  in  IBb*s  CP-67  [Aocencix 


4)  we  derive  the  general  software  implementation  of  third 
generation  VCS's.  A  close  scrutiny  of  typical  third  generation 


systems  leads  to  a  series  of  empirical  hardware  reouirements  for 
virtual izab  le  third  generation  architectures  [Section  3.2), 
Aoclication  of  these  rules  in  a  number  of  case  studies  indicates 
that  most  third  generation  systems  cannot  be  virtualized 
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(Appendix  ill  *  This  result  points  out  the  general  unsuitability 
of  third  generation  architectures  (for  virtualization)  and  the 
weaknesses  of  the  software  construction  technique.  Thus*  we  are 
forced  to  look  elsewhere  fo~  the  key  issues  3nd  solutions  in 
designing  virtual izab le  architectures  (Chapter  41. 

Other  material  presented  in  Chapter  3  incluoes  requirements 
for  hybrid  virtual  machines  (HVh)  (Section  3.3J,  ad  hoc 
improvements  to  third  generation  hardware  (Section  3.4),  software 
r eauirements  for  Type  II  (extended  machine  host)  virtual  machires 
(Sectior  3.51,  and  suggested  virtual  machine  software  crimitives 
(Section  3.5).  Additional  third  gereration  subjects  are  provided 
in  the  first  three  sopendices  which  deal  with  a  CP-67  tutorial 
[Aocendix  A],  case  studies  of  some  third  generation  machires 
(Appendix  P],  and  case  studies  of  seme  third  generation  Type  II 
operating  systems  (Aoperdix  C). 

Chapter  3  should  be  of  particular  interest  to  two  classes  of 
-Coders.  The  first  class  includes  these  individuals  running 
third  generation  systems  and  contemplating  the  introduction  of  a 
virtual  machine  facility.  The  various  empirical  rules  and 
studies  should  help  tc  cetermine  if  virtual  machines  are  possible 
on  the  reader’s  system  and  what  oractical  options  may  be  taken. 
The  second  class  of  reaoers  includes  computer  system  designers 
who  can  observe  how  relatively  insignificant  design  decisions 
have  rendered  most  tnirc  generation  machines  unvirtual izable.  It 
will  be  necessary  for  these  designers  to  avoid  such  errors  in 
designing  the  virtual  machine  support  for  the  Iv  generation. 

Chapter  5  gives  a  very  brief  summary  of  the  research  and 


X 


m 


L. 


**3S*&pgfP&Spfy  *-<‘7?  r-i**’-^^ •»  • 


g 


@1-- 


>.  . 
h 


M 


:a^sw» 


Page  13 


•joints  out  several  orcmisinc  areas  for  additional  study* 

The  final  material  of  tne  thesis*  Appendix  0,  is  a  glossary 
of  soire  of  the  terns  used  in  the  thesis.  In  addition  tc  the 
soecialired  terminology  for  virtual  machines*  e.g.  virtual 
machine  recursion  property*  a  number  of  general  tnough  not 
4 ide ly-krowr  terms*  e.g.  interior  decor*  are  also  defined. 
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Chroter 

2  is  divicel 

into  four  sections. 

The 

first  section 

introduces  r 

•lire  of  t  h  a 

basic  termirolcay 

that 

is  used  in  the 

thesis.  Sire 

a  virtual  c 3 

cuter  systems  have 

or  ly 

recertly  ccme 

intc  existence*  we  have  been  fcrcec  to  develcc  re»  terms  to 
-'ecresert  ice:*s  3bout  vCS*s.  The  accroach  take-  in  this  section 
is  somewhat  descriotiv*.  Later*  in  Sect  icr  4.2*  a  number  of 

these  rctiors  are  recast  ir  terms  cf  the  Vf S  mccel  cf  Chanter  4. 

Tr  the  second  sectian*  we  contrast  the  noticr  of  a  virtual 
T-ichir-£'  wiTh  a  number  cf  relatec  ccrcerts.  Next*  examcles  of 

ixistir.c  virtual  ccmcutar  systems  are  citec.  Ir  the  firal 

section.  cuhlisned  literature  relatirg  to  virtual  machines  is 
or  i  e  f  |  v  surveyed. 
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2.1  TERMINOLOGY  POP  VCS'S 


The  term  virtual  Ccnputer  System  has  been  loosely  aoplied  to 
3  r?n.-jo  cf  cifferent  coirouter  system  organizations.  Recently,  as 
noted  in  Chapter  1,  its  use  has  been  reserved  for  a  specific  type 
of  computer  system  entity. 


A  Virtual  Computer  System  is  a  hardware-software 
duplicate  of-  s  real  existing  computer  systeir  In  which  a 
statistically  dominant  subset  of  the  virtual 
orocessor*s  instructions  execute  directly  on  the  host 
processor  in  native  mode. 


There  are  two  parts  to  this  definition. 

Environment —  A  virtual  computer  systeir.  must  simulate  a  real 
existing  computer  systeir.  Programs  and  operating  systems  which 
run  or*  the  real  system  must  run  on  the  virtual  system  with 

identical  effect.  Since  the  simulated  machine  may  run  at  a 
lifferent  speed  from  the  real  one,  timing  deoencent  processor  and 

I/O  code  ray  not  perform  exactly  as  intersced.  This  is  very 

similar  to  the  limitation,  however,  that  characterizes 
interchangeability  of  programs  among  different  members  of  a 
“coirobt  it  I  e"  comouter  family,  such  as  between  tme  IBM  352/<i3  and 
the  35r/75 • 

Implementation--  host  instructions  being  executed  must  be 
processed  directly  by  the  host  CPU  without  recourse  to 
instruction  by  instruction  i nterpretat ion .  This  guarantees  that 
the  virtual  machine  will  run  on  the  host  with  relative 

efficiency.  It  also  compels  the  virtual  machine  to  be  similar  or 
identical  to  the  host,  and  forbids  tampering  with  tne  control 
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store  to  ado  an  entirely  new  order  code. 

The  VCS  ard/or  host  configurations  are  made  up  of  ore  or 
more  central  processors*  main  memory*  and  I/O  devices  (and 
oossibly  I/G  processors  too).  Usually  the  virtual  computer 
system  configuration  will  have  only  one  central  processor.  That 
orocessor  may  ne  called  a  KlClyaJ.  machi ne .  However,  the 
literature  often  informally  uses  virtual  machine  and  virtual 
computer  system,  \M  ard  VCS*  as  interchangeable.  Ws  shall  do 
I iKewi se. 

Sirce  a  VCS  is  a  hardware-sof tware  duplicate  of  a  "real 
existing  computer  system"  there  is  always  the  nctior  of  a  real 
computer  system*  PCS,  or  real  machine*  RM ,  whose  execution  is 
functionally  eauivalert  to  the  VCS. 

The  orooram  executing  on  the  host  machine  that  creates  the 
VCS  environment  is  called  the  Virtual  masliins  monitor  * 

CP-67  19*66*85],  which  is  ciscussed  below,  gives  the  VMh  the 
leneric  name  of  "control  program”.  Only  in  the  discussion  of 
CP-67  will  we  depart  from  the  use  of  VMM. 

The  VCS  differs  from  the  host  only  in  its  exact 
configuration,  e.g.  the  amount  of  memory  available  or  the 
oarticular  I/O  devices  attached,  fi  further  distinction  might  be 
in  the  orecise  characterist ics  of  the  virtual  processor.  The 
"implementation  reauiremenf"  in  the  VCS  definition*  above* 
implies  that  Ihe  virtual  processor  must  he  similar  or  identical 
to  the  host  processor.  If  they  are  not  identical,  ther  the 
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virtu?  I  machine  must  be  a  member  of  the  same  processor  family  as 
the  host.  For  example*  the  virtual  and  host  processors  might  be 
differert  models  of  a  compatible  crcduct  line*  e.e.  570/1F5  and 
360/3;'*  3&0/F7  and  360/65,  or  Data  General  Supernova  and  Nova. 
The  virtual  and  host  processors  might  even  be  the  same  model 
processor,  tut  only  differ  in  the  fullness  of  tie  instruction  set 
i  mp  I  emerited  *  e.g.  universal  instruction  set  vs.  standard 
instruction  set. 

This  distinction  may  be  captured  by  the  fcllowirg  two 
categories* 

Set  f-vjrtualizing  ( SV ) —  The  virtual  macnire  is  identical  tc  the 
most. 

Family-virtualizing  (FV)--  The  virtual  machine  is  a  memoer  of  the 
same  computer  family  as  the  host. 

In  a  sel f-virtua I izing  VCS*  the  functionally  equivalent  real 
computer  system  is  identical  tc  the  host.  Therefore*  we 
sometimes  call  the  host  machine,  the  real  machine.  We  then  say 
that  the  real /host  machine  has  a  v i rtua I i 2ah I e  architecture  [51]. 

If  a  VOS  is  se  I  f-virtua I izi^g,  ther  it  is  possible  to  run 
another  copy  of  the  VMM  on  the  VCS,  thus*  producing  another  level 
of  virtual  machines.  This  demonstrates  the  virtual  machine 
recursion  property  and  is  of  distirct  practical  importance  since 
it  allows  alterations  tc  be  made  to  the  VWM  running  or  the 
virtual  machine  and  permits  testing  the  VVM  in  the  rornral 
operating  environment.  [See  Section  4.2  for  the  develooment  of 
VM  recursion  in  terms  of  the  VCS  model.]  We  say  that  the  VMM 
defines  the  I gve I  of  recursion  or  virtualization.  If  a  VhM  is 
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rurring  or 

a 

level  r  VP*  then 

it 

onoduces 

a  level  n*l  Vk.  In 

aarticu  1  ar  * 

the 

real/host  machine 

is 

1  eve  1 

0. 

In  Figure  2-1*  the 

312K  virtual 

360/67  running  CP-67 

i  S 

1  ev  a  1 

1 

snc  its  two  virtual 

machines  are  level  2. 

l>£l££l  ££.•  L£Y££ 

The  notion  of  VCS  I  eve  1 s  should  not  be  confused  with  the 
corceDt  of  laveps  ir.  a  cotncuter  system.  As  will  be  discussed  in 
Chapters  3  and  4*  III  and  IV  generation  computer  systems  have 
structures  which  implement  layers  of  restricted  access  to  data  or 
instructions.  In  III  generation  systems*  the  two  Myers  are 
jsually  called  Vaster/S  I ave  moce  (suoervisor/prob I em, 
Executive/User*  etc.l.  In  IV  generation  systems*  the  layers  will 
likely  be  called  rings  [57*1061.  Since  a  layer  is  a  relationship 
oetweer.  two  parts  of  an  individual  VCS  ana  a  level  is  a 
relationship  between  two  VCS's,  they  are  somewhat  independent 
notions.  In  particular*  each  level  of  a  VCS  must  appear  to  have 
the  same  number  of  layers  as  the  real  machine.  However,  as  will 
be  seen  later*  [Section  4.31  software  implementation  cf  VM*s  have 
utilized  the  restricted  access  of  layers  to  simulate  the  effect 
of  levels.  This  is  merely  an  implementation  technique  and  not  a 
general  principle.  These  issues  are  discussed  furlher  in 
"Virtua I izeab le  Arch i tec tures"  151]  ana  Section  4.2.  Figure  2-2 
illustrates  the  relationship  between  levels  and  layers  for  the 
CP-67  VCS  shown  in  Figure  2-1. 
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TYPICAL  VCS  —  CP-67 
FIGURE  2-1 
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linlUai  ^ a stines  and  Related  Constructs 

The  implementation  recuirement  of  the  VCS  definition 
indicates  tnat  a  '‘statistically  dominant  subset  cf  the  virtual 
srocessor's  instructions  execute  cirectly  on  the  host  orocessor 
in  native  mode."  To  provide  functional  equivalence  between  the 
V CS  and  real  computer  system'  PCS*  those  instructions  which  co  r.ot 
execute  directly  must  be  simulated  on  an 
instruction-by-instruction  basis.  Thus*  the  •'performance**  of  the 
virtual  machine  is  bound  by  the  real  machine  ano  oy  s  "complete 
software  interpreter  machine*"  .  Rather  than  set  any  precise 
ner f ormance  objective  or  reouire  any  myopic  m3chiru.  dependent 
implementation  as  cart  cf  the  definition  of  virtual  machine,  we 
prefer  to  treat  VCS  as  a  hroac  class  of  potential  systems. 
[However,  for  a  particular  system  under  ccns i cerat i on  *  we 
normally  apply  the  term  VCS  if  the  direct  execution  subset  is 
maxima  I . J 

Thus*  the  "complete  software  interpreter  mschire"  is  rot  a 
VCS  since  no  instruction  of  the  CS7M  executes  directly.  Neither 
is  the  real  machine.  Although  all  instructions  of  the  PM  execute 
directly*  the  PM  is  not  a  hard war e-sof twar e  duplicate*  and  indeed 
it  requires  no  VMM. 

A  recently  distinguished  VCS  entity  is  the  hVfrCi 4  y jptual 
machine  hvm.  The  JjjfM  is  a  VM  in  which  a  I  I  supervisor  state*  e.g. 
Master  Xode  or  ping  0*  instructions  are  interpreted.  The  HVM  has 
oeer  found  to  be  a  useful  and  easy- tc-cons true t  artifact  where 
more  familiar  VCS  techniques  have  failed.  [See  Section  3.3  and 
Appendix  3.1 
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Ar  example  of  the  way  each  of  these  four*  constructs  might 
execute  a  oarticular  instruction  seouence  on  3  III  generation 
comruter  system  is  indicated  in  Figure  2-3.  Successive 
instruction  executions,  i.e.  time,  are  represented  from  left  to 
right  ard  the  two  states  shown  for  each  of  tne  four  constructs 
ire  software  interpretation  anc  direct  execution.  Thus,  the  real 
machine  is  always  shown  in  the  direct  execution  state  while  the 
virtual  machine  occasionally  traps  to  software  interpretation  for 
certain  instructions. 

Ii£§  I  lifie  11  If  MM 

The  implementation  requirement  specifies  that  Tost  VCS 
instructions  execute  cirectly  on  the  host.  It  coes  not  indicate 
now  the  VMM  gains  control  for  that  subset  of  instructions  which 
nust  be  interpreted.  This  may  be  done  either  by  a  program 
running  on  the  bare  host  machine  or  by  a  program  runninq  under 
sore  operating  system  on  the  host  machine.  In  the  case  of 
running  under  an  operating  system,  the  host  operating  system 
orimitives  may  be  usee  to  simolify  writing  the  virtual  machine 
monitor.  Thus,  two  additional  VCS  categories  arises 

T^CS  1“-The  VMM  runs  cn  a  bare  machine* 

Type  JJ--The  VMM  runs  or  an  extended  host  [53.751.  unoer  the  host 
operatinq  system. 

In  °ither  case,  the  virtual  machine  being  created  is 
eauivaient  to  the  bare  host  or  a  related  family  member. 

Tyre  I  (bare  host)  and  Type  II  (extended  nost)  VCS’s  are 
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illustrated  in  Figure  ?-*♦•  In  bo+h  Tyoe  I  an3  Type  II  VCS*  the 
1  MH  creates  the  VC S  ervirorment.  However,  in  a  Type  I  VC^»  the 
7Mk  on  a  bare  machine  must  perform  the  system’s  scheduling  snd 
(real)  resource  allocation.  Thus*  the  Tyoe  I  7H M  tav  include 
much  code  not  specifically  needed  for  a  VCS .  In  a  Tyoe  II  VCS, 
the  resource  allocation  anc  environirent  creation  functions  for 
V M* s  are  more  clearly  srlit.  The  operating  system  goes  the 
normal  system  resource  3llocatior  and  provides  s  standard 
extended  machine  environment.  The  VMM  program  runs  on  the 


exterded  machine  environment  and  oroduces 


pseudo-hardware 


environment.  (Another  interpretation  of  Tyoe  I  and  Type  II 
virtual  machines  is  presented  in  Section  4.2.1 

There  are  different  advantages  to  be  derived  from  using 
either  a  Type  I  or  Type  II  VCS.  If  an  installation  normally  purs 
one  particular  operating  system  and  most  users  would  prefer  to 
treat  the  system  as  an  exterded  machine*  then  a  Tyoe  II  VMh  rgy 
re  usee  to  provide  a  virtual  macnire  for  the  occasional  user  who 
may  desire  it.  He  might  wart  to  either  debuq  system  code  or, 
say,  run  some  foreigr  operating  system.  In  such  a  case,  the 
access  to  the  system  is  likely  to  be  more  convenient  for  the 
numerous  typical  users,  and,  probably,  more  efficient  since  there 
is  less  overhead  in  running  the  preferred  operating  system. 
[Another  approach  to  achieving  some  of  these  objectives  is 
neported  by  Parmelee  1931.1 

If  there  are  numerous  alien  operating  systems  tc  be  run  and 
there  is  no  such  thing  as  a  preferred  environment,  it  way  make 
sense  to  run  a  Type  T  VCS.  Of  course,  the  decision  to  choose 
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Type  II  over  Type  I  depends  also  on  the  existence  of  an  operating 
system  that  is  capable  cf  supporting  a  Type  II  system.  Another 
important  factor  to  consider  is  the  amount  of  work  required  to 
bring  up  either  system. 

Type  I  and  Type  II  V CS*s  are  two  cuadrants  of  a  tore  general 
tabular  structure  which  deserves  future  study.  See  Figure  2-5. 
The  two  rows  of  the  table  indicate  the  host  environment  on  which 
the  monitor  program  runs.  The  two  columns  indicate  the  target 
environment  produced  by  the  monitor.  A  Tyce  I  VMM  runs  on  a 
hardware  host  to  produce  a  hardware  target,  i.e.  virtual  machine. 
A  Type  II  VMM  runs  on  an  extended  host  to  produce  a  hardware 
target.  The  boxes  marked  £MM  stand  for  Extenaeo  M^gfrjne  Mori  tor. 
The  Type  I  EMM  is  a  conventional  operating  system.  The  Type  II 

EMM  runs  under  one  operating  system  to  produce  the  interface 
generated  by  some  other  operating  system.  Tyoe  II  EMM  can  be  a 
valuable  tool  for  transporting  software  when  a  VCS  is  not  used 
f  36 , 45 1 • 
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2.2  COMPARISON  KITH  RELATED  NOTIONS 


Because  of  the  recent  development  of  the  field  of  coirouter 
systems  architecture*  and  the  absence  of  a  generally  usee*  as 
well  as  agreed  upon*  terminology*  it  is  possible  that  the  notion 
af  a  virtual  machine  may  be  confused  with  certain  other  related 
ideas.  Tn  this  section*  some  common  misconceptions  regarding 
virtual  machines  will  be  cited  to  provide  some  additional  insight 
into  the  key  notions  involved  in  a  virtual  machine. 


2.2.1  Virtual  Hachire  vs.  Virtual  Memory 

Virtual  memory  refers  to  an  addressing  system  in  which  a 
logical  address  generated  by  a  program,  called  a  virtual  memory 
address,  is  mapped  into  some  other*  possibly  different,  physical 
•ne.rory  address.  Virtual  memory  may  be  implemented  in  numerous 
ways*  among  them  simple  relocation*  multiple  relocation,  paging* 
segmentation*  or  a  combination  of  these  methods  t383.  If  paging 
ar  segmentation  is  employed*  it  is  often  the  case  that  the  amount 
of  virtual  memory  exceecs  actual  physical  memory. 

Since  one  aspect  of  a  real  computer  system  is  memory*  the 
corresponding  attribute  of  a  virtual  computer  system  is 
custpmarily  called  virtual  memory  1511.  The  virtual!  memory  need 
not  be  larger  than  available  physical  memory,  although  it  may  be* 
if  implemented  on  a  machine  with  paging.  This  is  the  case  with 


CP-67  19*65*353.  The  virtual  memory  may  be  even  mapped 

identically  into  all  or  part  of  its  correspondingly  addressed 
physical  memory.  This  is  the  case  with  the  proposed  CP-65 
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[54,68],  experimental  IBM  360/39  C  7  i  3  ♦  or  Japanese  work  [501. 
The  permittee  virtual  memory  size  or  method  of  relccatior  (or 
not)  are  issues  which  affect  the  flexibility  of  the  use  of 
virtual  memory,  or  ease  of  performing  system  storage  allocation. 
These  issues  are  largely  tangential  to  the  notions  involvea  in 
virtual  machines. 

One  particular  connection  between  virtual  machines  and 
virtual  memory  concerns  the  interrupt  control  register  locations 
that  most  systems  have  fixed  in  low  physical  core.  Since  the 
memory  of  a  virtual  machine  is  functionally  eouivalant  to  the 
memory  of  a  real  machine,  virtual  interrupt  locations  in  virtual 
memory  must  have  the  same  apparent  effect  for  the  virtual  machine 
as  the  corresponding  real  locations  have  for  the  real  machine. 


2.2.2  Virtual  Machine  vs.  Virtual  Machine  Time-Sharing  System 

A  Virtual  Machine  Time-Sharing  System  (VMTSS)  is  a 
timesharing  system  in  which  £a£h  user’s  interface  with  the  system 
is  a*virtual  machine  (811.  It  is  possible  to  have  a  virtual 
machine  without  having  timesharing  or  a  VMTSS.  The  exoeri mental 
IBM  36T/30  [71]  or  the  Jaoanese  HITAC  B4P0  T 5 0 3  are  examples  of 
virtual  machines  running  on  a  non-ti mesh ar ing  system.  The 
virtual  360  under  UMMPS  I6G1  is  ar.  oxamole  of  a  Type  II  virtual 
machine  runring  under  a  (conventional)  timesharing  system  that  is 
not  a  VMTSS.  Possible  confusion  between  virtual  machines  and 
VMTSS’s  may  be  related  to  the  success  of  CP-67  which  is  a  VMTSS. 
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J? •  3  Virtu?!  *achire  vs.  Pseudo  Machine 


fl  oseudo-machine  1 99 1  *  also  called  an  extended  machine  [533 
or  a  user  machine  1753,  is  s  composite  machine  oroduced  through  a 
combination  of  hardwara  and  software*  in  which  the  machine’s 
aoperent  architecture  has  been  changec  Slightly  tc  make  the 
machine  more  converiert  to  use.  Typically,  these  architecture! 
changes  have  taken  the  form  of  removing  I/C  channels  3nd  devices* 
and  adding  system  calls  to  perform  I/O  ar.d  otrer  operations. 
These  system  functions  are  normally  invoked  by  transferring  to 
some  location  in  virtual  memory,  as  in  Muftirs  [86*921,  or 
issuing  a  Suoervisor  Call  (SVC)  or  Master  Moae  Entry  (MME)  ,  etc. 
instruction  with  suitable  address  code.  The  supervisor 
interprets  the  SVC,  executes  the  desired  function,  and  returns  to 
the  user. 

In  a  virtual  machine,  the  machine’s  apparent  architecture  is 
identical  to  a  rea I  machine’s  architecture.  The  visibility  of 
all  I/O  channels  anc  devices  is  ieft  intact.  There  are  no 
supervisory  functions  provided.  If  a  machine  wishes  to  perform 
I/O,  it  must  utilize  its  own  I/C  programs.  Indeed,  tne  system 
toes  not  provide  an  SVC  handler  for  the  virtual  machine. 

Possible  confusion  between  virtual  machine  and  pseudo 
machine  can  be  attributed  to  an  overuse  of  the  term  "virtual 
machine"  by  certain  authors.  Too  often  "virtual  machine"  has 
been  used  to  denote  a  non-hardware  "user"  machine  1111,112). 


2.2.**  Virtual  Machine  vs.  Emulated  Machine 

Emulation  is  a  technioue  that  has  beer  used  successfully  to 
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map  one  system  architecture  into  another  different  architecture 
(63,82,1101.  Emulation  at  the  Anterior  decor  interface  (nardwsre 
interface)  causes  the  layer  2ero  software  visibility  of  the 
native  system  to  (almost)  incluoe  vis*..i  ity  of  the  target 
system.  Similarly*  emulation  at  the  exterior  decor  interface 
(extended  machine)  extends  tne  software  visibility  of »  say*  layer 
l  to  include  software  visibility  of  layer  i  cf  the  target  system 
(511. 

Well-Known  examples  of  emulators  include  IE**s  1401  on  the 
36C/3G,  7D9i*  on  the  360/65,  and  DOS  under  OS  on  the  37:1/145.  In 
the  first  two  examples,  the  target  ar.d  native  interior  decors  are 
vastly  different.  This  reauires  the  emulator  tc  be  i to s anted 
via  special  additional  microcode,  and  the  processor  tU;'  operate 
in  this  non-native  mode. 

In  3  virtual  machire,  on  the  other  hand,  the  target  and  host 
are  either  similar  or  identical  hardware  machines.  If  a 
self-virtualizing  machine  has  an  emulator  implemented  on  it,  then 
it  should  be  accessible  to  the  virtual  machine  in  the  same  way 
that  it  is  to  the  real  one.  Thus,  if  there  exists  a  special 
instruction,  such  as  Oc  Interoretive  Loop  (OIL)  11101  which  puts 
the  machine  into  an  emulation  mode,  then  its  execution  by  a 
virtual  machine  shoulc  put  the  virtual  machine  into  emulation 

■node.  ( IBM  has  Just  announced  this  facility  for  vm/370  (67  1.1 
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There  are  relatively  few  examples  of  virtual  machines  that 
nay  be  cited.  We  will  show  later  [Section  3.21  that  this  is  cue 
to  the  general  unsuitability  of  contemporary  designs  and  hence 
the  significant  requirements  placed  on  a  computer  system  that  is 
to  support  a  sc.tware  construction  of  virtual  machines. 
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2.3.1  IBM  Pesearch  M4W44X--  Type  T  FV  (Family-virtualizing) 

The  IBM  M44  [91*1031  was  a  hignly  modified  (II  generation) 
IBM  7C-44*  extended  tc  include  paging  and  a  number  of  ooeratioral 
modes*  such  as  Prob  I  etr/Superv  i  sor  *  Location  Test/No  Location 
Test.  and  Mapoing/Nc  Mapping.  Through  the  use  of  an  operating 
system*  called  MOS  (Modular  Operating  System),  it  oroduced  a  set 
of  virtual  machines  called  44X*s.  IThese  44X's  are  not  virtual 
machines  in  the  strictest  sense  since  they  are  slightly  different 
from  real  7044's.l  A  44X  operating  system  runs  on  each  44X  and 
orovides  the  customary  services  found  on  most  operating  systems. 
The  M44/44*  system  was  very  important,  in  the  history  of 
computing*  for  trying  out  a  number  of  innovative  ideas,  among 
them*  virtual  machines. 

2.3.2  IBM  Cambridge  Scientific  Center  CP-40 —  Tyoe  I  FV 

CP-40  [3*27*59)*  the  forerunner  to  CP-67*  was  constructed  on 
an  IBM  360/40  that  was  soecially  mccified  through  the  addition  of 
an  associative  memory  paging  box.  This  system  was  a  VMTSS  and 
supported  fourteen  virtual  360's.  These  350's  were  standard 
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360’s  anc)  did  not  contain  any  of  the  special  mooi  f ications  cf  the 
hoot  moael  4f .  The  virtual  36f’s  were  able  to  run  numerous  360 
operating  systems*  including  the  standard  OS/363  [66*64]  and  CMS 
[85].  a  conversational  system  which  was  developed  at  Cambridge 
Scientific  Center  specifically  for  use  with  CP-40.  The  system 
suffered  from  the  composite  limitations  of  the  3S9/4P’s  processor 
speed*  a  modest  amount  of  real  core*  and  a  slow  disk  for  paging. 
However.  CP-40  die  serve  to  demonstrate  the  viability  of  the 
virtual  machine  notion.  Furthermore*  it  provided  a  VMTSS*  and 
with  CMS*  a  convenient  conversational  system  for  the  360  at  a 
time  when  no  other  gereral  purpose  timesharing  system  for  that 
series  of  machines  was  working.  Irdeed*  success  with  this 
project  led  directly  to  the  larger  system*  CP-67. 


2.3.3  ISM  Cambridge  Scientific  Center  CP-67 —  Tyoe  I  FV 

CP-67  [9*65,85]  was  begun  as  an  experimental  system  in 
conjunction  with  MIT  Lincoln  Laboratory  in  January,  1967. 
Initially,  the  CP-40  system  was  modified  slightly  to  suooort  the 
different  memory  relocation  system  found  on  the  360/67. 
Subseauent I y,  the  entire  CP-67  system  was  rewritten.  CP-67  is  a 
VMTSS  which  runs  on  the  369/57  and  supports  oetween  thirty  and 
forty  users,  ’t  is-.  by  far,  the  most  widely  known  virtual 
machine  system.  The  virtual  36C*s  produced  under  CP-67  have  been 
very  successful  in  runnirg  a  number  of  different  360  operating 
systems.  These  include  (but  are  not  limited  to)  CS/360  (in  many 
versions)*  DOS,  DOS-AFL,  CMS*  LLMFS  185).  The  virtual  machines 
have  been  used  in  telecommunications  acol ications.  The 
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University  of  Grenoble  has  even  rur.  the  IBP  CS/ASP  (Attached 
Support  Processor)  system*  connecting  a  real  360/40  to  s  virtual 
360  C  9 ) « 

The  orincipal  limitations  of  the  virtual  machines  produced 
by  CP-67  are  that  a  small  set  of  programs  may  not  work  correctly. 
These  are: 

(1)  Programs  which  depend  on  precise  execution  times? 

(2)  Programs  which  deoeno  on  the  relationship  between 
processing  time  and  input/output  time* 

(3)  Proqrams  which  depend  on  precise  real-time*  and 

(4)  Programs  which  certain  set f-moai tying  channel 
programs • 


The  first  two  restrictions  are  identical  with  those  to  insure 
compatibility  between  two  different  member  processors  of  the  360 
family  [691. 


3.3.4  Extension  of  CP-67  to  support  virtual  360/67--  Tvoe  I  SV 

In  May*  1969*  the  author  designed  a  slight  extension  to 
CP-67  to  support  a  relatively  efficient  i mo  I ement3t ior  of  a 
virtual  360/67  (with  virtual  relocation)  £541.  Kith  slight 
modifications*  this  design  of  a  virtual  360/67  was  implemented  in 
the  fall  cf  1969  by  F.  Furtek.  Independently  cf  this  work*  A. 
Auroux  st  the  IBM  Cambridge  Scientific  Center  also  desigred  a 
virtual  360/67  extension  to  CP-67  (951.  'The  two  cesigns  are  very 
similar  and  the  Auroux  implementation  has  pecome  P3rt  of  the 
standard  (IFM  distributed)  CP-67  system. 

The  virtual  360/67  has  been  limited  to  a  64-bit  addressing 
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nooe  single  processor  machine.  Kith  this  caoability,  various 


360/67  ooerating  systems  have  been  run*  including  CP-67  end 


TSS/760  £951.  That  is*  CP-67  has  been  run  under  itself.  In 


fact,  this  chain  has  been  tested  to  a  depth  of  at  least  four. 


The  orircipal  additional  limitation  on  the  operation  of  the 


virtual  360/67  over  a  standard  virtual  360  (under  CP-67)  is  that 


orograms  that  dynamically  alter  the  contents  of  segment  cr  page 


tables  may  not  execute  as  intended.  The  results  may  or  may  net 


oe  identical  with  a  real  360/67.  Ir  a  real  machine,  the  cortents 


of  the  associative  registers  may  affect  the  result  and  hence 


results  may  also  be  unpredictable. 


2.3.5  Virtual  360  under  UMhPS—  Type  II  FV 


The  virtual  36C  under  UMMFS  Ik, 60]  is  a  Type  IT  virtual 


machine  system.  UMMPS  is  a  conventional  mul t i programming  system 


which  was  written  at  The  University  of  Michigan  to  run  on  a  dual 


orocessor  360/67.  The  University  of  British  Columbia  has  trade  a 


few  modifications  to  UMMPS  to  support  a  virtual  369.  These 


modifications  have  taken  the  form  of  some  additional  supervisor 


call  facilities  to  perform  some  crucial  operations  in  dispatching 


the  virtual  machine.  Apparen*  exceptional  conditions  occurring 


in  the  virtual  machine  are-  reflected  by  the  UMMPS  supervisor  bacK 


to  a  user  program  which  performs,  for  example,  the  simulation  of 


privileged  instructions.  Virtual  (standard)  360*s  are  the  only 


virtual  machines  supoorted  by  this  system,  a  I ‘hough  other  user 


interfaces  besides  the  virtual  machine  interface  are  provided 


(such  as  MTS  or  a  conventional  batch  system'.  Extensions  to 
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'JMMFS  to  support  virtual  36C/67*s  as  well  as  the  standard  36G*s 
are  being  designed. 

2.3.6  Japanese  Electrotechnical  Lab.  HITAC-R403--  Type  II  F V 

The  HIT AC-840C  is  the  Japanese  manu f ac tuned  version  of  the 
RCA  Spectra  70/45.  It  is  a  conventional  third  generation 
computer  without  a  memory  rapping  system*  siTilar  to  the  IRM 
System/360.  As  an  aid  in  debugging  ETSS  1491*  3  new  tire-sharing 
system  for  the  HITAC-S4QC*  a  Type  II  virtual  machine  was 

constructed  on  the  existing*  vendor-supplied  operating  system* 
TOS/TOOS  I5P1.  The  virtual  machine  system  is  very  similar  to  the 
virtual  360  constructed  on  UMMPS*  One  interesting  escect  of  this 
effort  was  the  attempt  at  simulating  various  virtual  I/O  devices 
that  did  not  have  exact  physical  counterparts  (such  as  a  display 
scope).  This  was  done*  of  course*  since  the  virtual  machine  was 
being  used  entirely  to  debug  supervisory  code  for  ETSS  and  not 
for  rurring  production  jobs  under  it. 

2.3.7  Hardware  Modified  IBM  36T/30--  Type  I  c  V 

A  virtual  machine  system  has  beer  constructed  on  an 

experimental  363/3A  through  a  combination  o*  software  and 

hardware*  i.e.  microprogramming  modifications  [711*  These 
nodi f ications*  wnich  have  been  introduced  in  order  to  simplify 
virtual  machine  constructions  are  used  to  im-oiemept  s  special 

"•monitor”  mode*  and  to  relocate  the  interrupt  control  area  from 
the  regular  control  program  area  in  physical  page  0.  The  system 
is  not  a  VMTSS  anc  only  supports  one  virtual  machine.  Its 
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orirciole  use  is  in  monitoring  and  evaluating  tne  performance  of 
existing  operating  systems  such  as  Operating  System/360  (OS/360), 
and  the  Basic*  Disk,  and  Tape  Operating  Systems  (EOS*  DCS* 


T0S/36D  . 


2.3.8  HIT  Project  MAC  PCP-1C  ITS--  Type  II  SV  HVM 

As  will  be  seen  in  Chapter  3*  it  is  not  oossible  to 
implement  virtual  machines  or  the  OEC  PDP-19  using  conventional 
third  generation  software  techniaues.  Recently*  the  author  and 
S.  k.  Galley  have  introduced  the  notion  of  a  hybrid  virtual 
machine  for  the  POP-13  1561.  With  this  technique*  all 
execut ive-mcde  instructors  are  irterpretsd  while  all  user-mode 
instructions  execute  directly.  The  MIT  Project  *AC  Dynamic 
Mode  I irg-Computer  Graphics  POP-10  has  provided  an  ideal 
environment  for  this  i mo  I ementat  ion •  The  ITS  (Incompatible 
Timesharing  System)  provides  the  requisite  features  to  support  3 
Type  II  (extended  machine  host)  VCS.  Furthermore,  tre  hardware 
configuration  is  trlessed  with  sufficient  cere  and*  at  times* 
enough  CPU  cycles  to  make  it  possible  to  run  the  HVM«  (See 
discussion  in  Appendix  C. 1 


2.3.9  Grenoble  Monitor  System  for  Standard  360/kH 

GMS*  Grenoble  Monitor  System,  158)  was  designed  by  the  TBM 
Grenoble  Scientific  Center  to  provide  multiole  369*s  on  a 
standaro  IBM  365*  i.e.  one  without  relocation.  The  major 
objective  of  the  system  was  the  concurrent  suppert  of  3  limited 

number  of  virtual  machines*  each  running  CMS*  the  Cambridge 
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Monitor  System*  Since  3  standard  360  does  not  orovide  a  memory 
relocation  system,  the  store  and  ifi ±Qh  memory  protection  system 
must  be  usee  in  virtual  machine  construction.  ISae  Appendix  B.l 
However,  because  of  difficulties  in  installing  fetch-protect  cn  a 
Puropear  model  40,  the  actual  implementation  had  tc  compromise 
the  pure  360  hardware  definition  of  the  virtual  machines.  This 
led  to  a  system  in  which  modifications  were  mace  to  CHS  to  allow 
it  to  run  in  this  “enhanced**  environment. 

2.3.10  IBM  VM/370 

In  July,  1972,  I8M  announced  a  virtual  machine  facility  for 
various  models  of  the  System/370  C67].  This  system  seems  to  be  a 
direct  descendant  of  CP-67.  When  delivered,  VM/37C  will  be  the 
first  f ormsl ly-supported,  commercia I  ly-of ferred  VCS  available 
from  any  manu facturer . 
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The  early  literature  on  virtual  machines  dealt  almost 
exclusively  with  the  imo I ementati on  mechanisms*  policies*  or 
applications  of  ore  particular  virtual  comouter  system.  The 
treatment  of  policies*  e.a.*  scheduling  algorithms*  pagt 
replacement  algorithms*  in  these  paoers  involves  largely  the  same 
consi aer at i cns  as  in  non-virtual  machine  timesharing  or 
mu  I  tiprogr smming  systems.  Therefore*  they  will  not  be  reviewed 
nere.  Many  of  these  papers*  particularly  those  devoted  to  memory 
management  appear  in  the  annotated  bibliography  of  Farmelee  et  al 
1951.  Since  the  tnesis  is  concerned  with  levelocing 
architectural  principles  ana  mechanisms*  we  will  rot  review  the 
virtual  machine  aoplications  papers  either.  However*  some  of  the 
■more  significant  oapers  3re  listed  in  the  bibliography. 

The  virtual  machine  implementation  and  machanism  papers  go 
■jack  to  an  early  unpublished  pacer  by  Sayre  [101,133]  which 
lescribes  the  IBM  M44/44K  work  and  its  implications.  Perhaps  the 
first  virtual  machine  report  to  get  wider  circulation  was  the 
jescriotion  of  OP-40  produced  by  the  IBM  Cambridge  Scientific 
Center  in  1966  [31. 

With  the  arrival  of  CP-67*  a  number  of  oaoers  were  oublish.ed 
which  cescribe  the  mechanisms  used  to  create  virtual  machines. 
These  papers  include  Field  [47]  and  Aurcux  and  Hans  (in  French) 
[91  which  are  the  most  complete.  Later  papers  by  IB*  authors, 
a.g.  Meyer  and  Seawright  [051*  repeat  much  of  the  same  oublished 
materia  I . 
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Two  early  papers  deserve  particular  note.  Fuc^i's  paper 
1501  reports  on  the  i  mp  lamentation  of  the  first  Type  II  VMM  and 
also  gives  insightful  suggestions  for  hardware  "improvements"  to 
facilitate  virtual  machine  construction.  Keefe's  paper  171] 
reports  on  some  hardware  modifications  attempted  with  an  IBM 
360/33  in  order  to  simplify  production  of  a  non-self-virtualizing 
J  MM. 

Among  the  author's  early  papers  are  the  first  caoers  that 
discussed  se If-virtua I izirg  systems  and  the  development  of  an 
empirical  basis  for  deciding  which  machines  are 
self-virtualizing.  "Virtual  Machire  Systems'*  145]  recorts  on  the 
design  of  a  virtual  360/67  under  CP-67  and  a  virtual  360/65  under 
"CP-65"  (which  runs  on  a  standard  350/65),  This  same  report  and 
"Hardware  Reauirements  for  Virtual  Machine  Systems"  15?]  contain 
the  first  published  analyses  of  the  problems  involved  in 
implementing  a  VMM  on  a  specific  class*  i.e.  third  generation 
machines,  rather  than  an  individual  computer  system. 

Recently,  there  has  been  increased  interest  in  VCS's.  At 
the  (September)  1971  IEEF  International  Computer  Society 
Conference,  the  author  organized  a  session  of  saoers  153,94,113) 
3hd  discussion  devoted  solely  to  VCS's.  Cne  cf  these  papers, 
written  by  the  author,  "Virtual  Machines*  Semantics  and  Examples" 
1531  proposed  some  of  the  terminology  introduces  in  Chaoter  ?. 

At  the  1972  ACM  International  Computing  Syrposium  at  Venice, 
three  papers  specifically  related  to  VCS's  appeared.  In  "Virtual 
Input/Cutout  in  a  Virtual  Environment"  18),  Ancilotti,  Cavina  and 
Lijtmaer  of  Pisa  investigate  the  possibility  cf  allowing  an  I/O 
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channel  to  execute  s  program  allocated  in  virtual  memory.  "Is 
Supervisor-state  Necessary?"  1761  by  Lauer  ard  Snow  cf  the 
University  of  Newcast le-uoon-Tyne  is  a  particularly  insightful 
and  well-written  caper  proposing  the  organization  of  a 
self-virtualizing  computer  system  based  upon  a  relocation  and 
bourds  form  of  address  relocation  and  no  supervisor  state.  As 
will  be  discussed  in  Section  4.8.  this  design  is  3  special 
sub-case  of  the  hardware  virtualizer  of  Section  4.5  with  f  =  R-B 
and  4=iaentity  map. 

The  third  oa per  was  “Virtua I izeab I e  Architectures"  (511  by 
U.O.  Gsgliardi  3nd  the  author.  This  paper  was  the  first  paper  to 
suggest  the  need  for  firmware  support  for  virtual  machines  in  a 
complex  comcutinq  environment.  The  design  that  emerged,  called 
the  Venice  Proposal  (VP)  in  this  thesis,  features  a  hardware 
Virtual  Machine  Identifier  (VMIO)  Register  to  specify  the  active 
virtual  machine.  It  was  the  work  with  U.O.  G3gliardi  on  the  VP 
that  lea  the  author  directly  tc  the  more  general  model  of  a  VCS 
oresented  in  Section  4.2.  In  Section  4. 4.  the  VP  is  presented 
and  interpreted  as  a  special  case  of  the  VCS  model. 

A  number  of  other  p3pers»  not  specifically  related  to  VCS’s. 
have  affected  the  author’s  thinking  during  the  course  of  this 
research.  Some  of  these  papers  ciscuss  hierarchical  operating 
systems,  layered  protection  structures,  or  ceferred  resource 
oinding.  The  early  paper  by  Dennis  and  Van  Horn  (1.40)  describes 
an  hierarchical  operating  system  and  suggests  primitives  for 
interprocess  control.  The  caper  by  Oijkstra  (41)  describes  a 
view  of  a  system  seen  as  a  layered  structure.  The  paper  by 
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Schroeder  and  Saltzer  [1C61  proposes  a  hardware  mechanism 
sufficient  to  support  a  layered  system  structure*  Evans  and 
LeClerc  [46*771  describe  a  hardware  addressing  structure  in  which 
addresses*  e.g.  segment  numbers*  are  assigned  relative  to  each 
individual  procedure  call.  Finally,  Schell's  doctoral  thesis 
tlO<«]  discusses  the  relationship  between  logical  and  physical 
resources  and  the  mechanisms  for  electrical  binoirg. 
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CHAPTER  3. 

PRINCIPLES  FOR  III  GENERATION  VIRTUAL  COMPUTER  SYSTEMS 
Inadecuacy  of  Current  Designs 


3.0  PLAN  OF  CHAPTER  3 


Chapter  3  develops  orinciples  for  third  generation  virtual 
computer  systems.  The  aooroach  adopted  is  somewhat  empirical  and 
relies  on  a  scrutiny  of  existing  III  generation  hardware  and 
software  systems.  The  chapter  is  divided  irtc  four  subject 
areas*  (1)  Characteristics  of  III  generation  hardware.  (2) 
Harcware  reauirements  for  third  generation  virtual  computer 
systems.  (3)  Software  reauirements  for  Type  II  third  generation 
VCS  *s»  srd  (4)  Conclusion. 

In  the  first  section,  we  summarize  relevant  architectural 
characteristics  of  III  generation  systems. 

Section  3.2  generalizes  the  virtual  machine  construction 
jsec  in  CP-67  to  derive  a  set  of  empirical  hardware  requirements 
for  determining  if  a  third  generation  architecture  is 
virtua  I  izab  le.  [Appendix  A  provides  a  brief  tutorial  on  CP-67.) 
The  rules  are  applied  to  a  number  of  representative  III 
generation  machines  and  the  results  are  surnmarizec  in  a  table. 
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"improvements"  are  considered  to  make  more  third  generation 
machines  virtua I izable. 

Section  3.5  develops  a  series  of  empirical  ooerating  system 
reauirements  for  determining  if  an  operating  system  on  a  third 
generation  rachine  will  support  a  Type  II  (extended  machine  host) 
VMh.  The  rules  are  applied  to  a  number  of  r eoreser tat ive  III 
generation  operating  systems  and  the  results  are  summarized  in  a 
table.  lAppendix  C  provides  the  detailed  analysis  of  these  case 
stucies.l  In  Section  3.6*  we  propose  an  example  of  a  set  of 
idealized  operating  system  primitives  to  support  Type  II  virtual 
machines . 

Section  3.7  reviews  some  of  the  results  cf  the  chapter  and 
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3.1  CHARACTERISTICS  CF  III  GENERATION  COMPUTER  SYSTEMS 

One  of  the  most  significant  architectural  innovations  of  III 
generation  computer  systems  is  the  existence  of  two  orocessor 
modes  cf  operation.  These  two  moces»  variously  called  supervisor 
and  problem,  executive  and  user,  or  master  and  slave  provide  two 
degrees  of  privilege  for  programs.  Thus,  operating  system 
programs,  when  executing  in  master  mode,  may  typically  invoke 
certain  special  instructions,  while  user  programs,  operating  in 
slave  mode,  may  not.  These  instructions  normally  control  such 
critical  system  features  as  the  input/output  facilities  and  the 
setting  of  privilege  itself.  Attempts  by  programs  in  slave  mode 
to  issue  privileged  instructions  are  normally  prevented  in  one  of 
several  ways. 

Another  arch i tectura I  characteristic  of  III  generation 
systems  is,  often,  the  presence  of  a  supervisory  call 
instruction.  Since  a  user  program  is  Drohibitec  from  issuing 
privileged  instructions,  the  operating  system  normally  provides  a 
means  for  users  to  call  upon  it  for  these  services,  f.-.try  to  the 
operating  system  is  usually  via  a  special  instruction,  such  as 
Supervisor  Call  (SVC,  RRM,  UUO,  MME)  which  traos  to  a  oarticular 
location  in  the  monitor.  The  monitor  checks  the  reauest, 
performs  (if  permitted)  the  desired  function  ana  returns  to  the 
user  program. 

An  additional  characteristic  of  the  III  generation  computer 
system  concerns  the  means  cf  addressing  memory.  Usually,  but  not 
always,  some  kind  of  relocation  and/or  protection  system  is 
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orovided.  The  memory  relocation  nay  take  the  form  of  simple 
relocation  and  bounds  (R-3) »  oaging.  or  even  segmentation  and 
oaging.  Host  of  these  systems  provide  an  absolute  addressing 
mode  when  the  memory  relocation  system  is  disabled.  This  usually 
occurs  when  master  mode  is  entered. 

TII  generation  I/O  systems  usually  communicate  with  the  CPU 
via  asynchronous  interrupts  which  are  affected  by  an  interrupt 
control  area  in  low  Physical  core.  Interval  timers  are  set  and 
notify  the  CPU  similarly.  ' 

Denning  137]  has  recently  delineated  the  extent  of  III 
generation  systems  by  uncovering  the  existence  cf  a  "L3te  Third 
Generation."  Since  his  distinction  between  III  and  Late  III  is 
often  based  on  facilities  of  the  operating  system  software*  we 
need  not  distinguish  them.  Virtualisation  is  concerned  with  the 
software  visibility  of  the  interior  decor,  not  of  any  operating 
system  features.  Later  in  Chapter  4.  we  take  up  the  likely 
characteristics  of  IV  generation  architectures.  Since  many  of 
these  same  "Late  Third  Generation"  facilities.  e.g.»  interprocess 
communication,  are  now  present  in  the  IV  generation  interior 
leccr.  virtualization  must  consider  them.  ISee  Section  4.1.1 
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3.2  EMPIRICAL  HARDWARE  REQUIREMENTS 


CP-67  has  suggested  a  general  approach  for  constructing 


virtual  machines  on  other  conventional  III  generation  systerrs. 


In  this  section  we  will  explore  the  application  of  the  CP-67 


virtual  machine  construction  technique  and  see  for  which  III 


generation  systems  it  is  appropriate.  Appendix  A  provides  a 


tutorial  or.  CP-67  and  describes  the  maos  which  it  employs.  The 


Key  point  in  the  simulation  of  a  (III  generation)  virtual 


orocessor  is  that  sirce  the  host  and  virtual  machine  are 


identical  (or  similar),  the  instructions  of  the  virtual  machine 


may  be  executed  directly  on  the  host.  However,  because  of  the 


virtual  machine  mapping  construction  it  is  necessary  to  prevent 


certain  of  the  instructions  from  executing  directly  cn  the  host. 


These  instructions,  if  executed  directly  by  a  VM  can  cause 


serious  difficulties  in  i nterpretat icn  or  control.  ISee  Appendix 


A.]  Any  instruction  whose  direct  execution  cannot  be  tolerated  is 


termed  a  sens i t iye  instruction.  Then  the  Key  to  implementing  a 


virtual  machine  on  III  generation  systems  is  to  orovide  complete 


functional  equivalence  with  a  real  machine  without  allowing  the 


direct  execution  of  sensitive  instructions. 


The  scheme  adopted  relies  on  a  software  mapping  and  permits 


only  the  VMM  to  run  in  supervisor  state.  The  virtual  machine, 


itself,  is  run  in  problem  state.  Therefore,  if  the  instructions 


that  are  considered  sensitive  are  also  privileged,  their 


attempted  execution  by  the  virtual  machine  (ir  problem  state) 


will  cause  a  trap  to  the  VMM.  If  there  are  some  instructions 
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that  are  considered  sensitive  but  are  not  orivileqed 
instructions*  then  it  may  be  necessary  to  resort  to  massive 
interpretation  in  order  to  preserve  the  integrity  of  the  virtual 
machine. 

What  are  the  instructions  that  are  corsiderea  sensitive  ir  a 
virtual  machine  system?  Certainly  any  instruction  that  attempts 
to  charge  the  mode  of  the  virtual  machine.  As  stated  above*  in 
order  to  prevent  a  virtual  machire  from  gaining  control  cf  the 
host  machine*  the  virtual  machine  must  be  run  in  problem  state. 
Moreover  the  virtual  machine  should  not  be  permitted  to  out  the 
host  machine  into  supervisor  state  in  such  a  way  3s  tc  allow  the 
virtual  machine  to  execute  any  privileged  instructions.  In 
order  to  mairtain  a  compatibility  between  the  virtual  machine  and 
its  real  counterpart*  the  VMM  must  provide  a  virtual  supervisor 
state.  See  Figure  3*1.  In  this  state*  the  virtual  machine’s 
orivileged  instructions  are  simulated  by  the  VMM.  In  addition, 
the  virtual  machine  should  not  be  able  to  charge  its  state  from 
virtual  supervisor  to  virtual  problem  state  without  a  orivileged 
instruction.  Since  both  of  these  states  are  actually  run  on  the 
-tost  machine  as  problem  state*  the  VMM  must  be  informed  directly 
or  it  would  have  no  way  of  knowing  that  a  victual  state  change 
has  occurred.  Furthermore*  it  is  not  sufficient  to  merely 
orevent  the  virtual  machine  from  changing  the  state  of  the  host 
machine,  it  must  also  be  prevented  from  referencing  it.  Since 
the  host  machine  will  be  in  problem  state  when  the  virtual 
machine  thinks  it  is  in  supervisor  state,  the  answer  to  the 
question  "What  state  am  I  in?"  may  give  an  inappropriate  result. 
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Conseouert I y  the  instructions  which  reference  state  information 
it  well  as  those  that  actually  change  it  must  be  orivileged 
instructions.  In  machines  in  which  this  is  not  true,  if  the  V MM 
cannot  perform  some  kind  of  cre-'i  ntarpretat  ion,  a  virtual  state 
charge  fright  occur  without  its  knowledge.  As  will  he  seen  in 
Section  3.3,  the  hybrid  virtual  machine*  HVM*  provides  another 
solution  to  this  dilemma. 

Another  problem  arising  from  the  fact  that  virtual  machines 
in  virtual  supervisor  state  must,  in  fact,  oe  run  in  physical 
problem  state,  concerns  the  interpretation  or  lack  of 
interpretation  of  certain  bits  in  the  instruction  wo.  j.  Certain 
machines  use  an  additional  bit  in  the  instruction  word  only  when 
in  supervisor  state.  Other  machines  use  additional  bits  in  the 
address  portion  of  the  instruction.  These  differences  in 
instruction  execution  in  problem  ^nd  supervisor  state  may  be 
difficult  to  resolve  without  resorting  to  instruction- 
bv-instruct ion  simulation.  [See  Appendix  P.l 

In  addition  to  changing  cr  referencing  the  sta'j  of  the  host 
machine,  the  virtual  machine  must  be  prevented  ircm  tampering 
with  or  looking  at  certain  sensitive  registers  and  cere 
locations.  For  example,  because  the  VMM  must  keeo  tr3ck  cf  the 
~ea I  interval  timer  fer  all  machines,  the  virtu3l  machine  must  be 
prevented  from  referencing  the  real  clock.  Since  in  seme 
machines  the  clock  is  a  register  or  a  fixed  location  in  core, 
either  the  instruction  tnaT  references  it  must  be  privileged  or 
some  method  must  be  found  for  orotecting  i t  through  the  use  of 
the  protection  or  relocation  system.  In  order  to  preserve  the 


virtual  machine  notion,  the  VMM  must  maintain  a  virtual  clock  for 
the  virtual  machire  in  some  other  part  of  core.  Some  other 
sensitive  dedicated  core  locations  are  the  various  interrupt 
registers  that  are  automatically  accessed  by  the  hardware  in  the 
evert  of  an  interrupt.  Since  the  contents  of  the  interrupt 
locations  are  usee  automatically  by  the  hardware*  their  values 
may  be  considered  part  of  the  soeci f ication  of  the  state  of  the 
nachine.  Therefore*  the  instructions  that  access  these  values 
are  sensitive  and  should  be  privileged.  If  they  are  not*  some 
other  means  for  protection  and  alerting  must  be  employed. 

Other  sensitive  instructions  are  those  involving  the  storage 
orotection  system  and  address  relocation  system.  In  orcer  to 
protect  the  supervisor  and  (possibly)  other  virtual  machines  from 
an  errant  virtual  machire,  the  virtual  machine  may  never  have 
access  to  any  location  that  . is  not  in  its  virtual  memory.  This 
may  be  accomplished  by  making  all  storage  protection  instructions 
orivileged,  or  oermitting  their  effect  only  within  the  aedress 
space  defined  by  the  address  relocation  system..  Ij  there  is  no 
relocation  system  and  absolute  addresses  are  generated,  than  the 
storage  orotection  mechanism  must  be  used  to  set  the  virtual 
memory  size,  protect  dedicated  lower  core  locations  from  access* 
and  protect  the  supervisor.  In  this  case,  the  storage  orotection 
instruction  must  be  privileged  to  oermit  the  supervisor  to  manage 
the  protection  keys  for  the  host  machine.  If  the  host  machine 
nas  a  storaqe  relocation  system  (either  relocation  registers* 
oaging,  segmentation,  or  a  combination  of  these)  ther  it  must  be 
insulated  from  the  action  of  the  virtual  machine  by  making  all 
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instructions  referencing  the  relocation  system  privileged.  If  it 
is  cesired  tc  have  a  virtual  relocation  system  which  makes  use  of 
the  host  machine's  relocation  hardware  then  the  effect  of 
melccation  setting  instructions  must  be  simulated.  This 
simulation  includes  translating  virtual  memory  locations  tc  real 
core  addresses. 

Input-output  instructions  are  always  sensitive.  It  is 
necessary  that  they  be  executed  only  cy  the  supervisor  in  order 
to  isolate  core  from  secondary  storage,  protect  secondary  storage 
from  virtual  machines,  and  enable  the  efficient  scheduling  of  I/O 
tasks  for  the  host  machine.  Furthermore,  the  VVM  may  be  recuired 
to  translate  virtual  device  addresses  into  their  real  equivalents 
before  I/O  transmission  may  take  place.  If  the  notification  of 
the  I/C  instruction  does  not  occur  before  the  I/O  operation  is 
started,  data  may  be  lost  or  physical  movement  may  be  needlessly 
initiated.  Thus  it  is  important  to  trap  the  I/C  instruction  as 
soor  as  possible.  ft  privileged  I/O  instruction  is  the  best 
solution. 

Thus,  we  are  able  to  state  the  following  empirical  hardware 
rules  which  govern  whether  a  virtual  machine  monitor  may  be 
implemented  on  a  conventional  third  generation  computer  system. 


(1)  The  irethod  of  instruction  execution  of 
non-pri v i  I eged  instructions  in  both  supervisor  and 
problem  state  must  be  roughly  eauivalent  for  a  large 
subset  of  the  instruction  repertoire. 

(?.)  A  methoo  cf  protecting  the  supervisor  (anc  any 
other  virtual  machine*  if  there  is  mu  I  ticroqrarrming) 
from  the  active  virtual  machine  must  be  available. 
This  may  be  accomplished*  for  ex3mcle,  through  a 
protection  system  or  an  address  translation  system. 

13)  A  method  cf  automatically  signalling  the  supervisor 
when  tne  virtual  machine  attempts  to  execute  a 
sensitive  instnuction  must  be  available.  The  trap  must 
not  cause  urrecoversb I e  errors.  It  must  then  he 
possible  for  the  supervisor  to  simulate  the  effect  of 
the  instruction.  Sensitive  instructions  includet 

a.  These  instructions  which  alter  cr  cuery  the 
state  of  tne  machine*  e.g.  "Ts  it  in  supervisor 
state  or  oroblem  state?*',  "Is  it  in  relocate  mode 
or  not?" 

b.  Those  instructions  which  altar  or  suery  the 
state  cf  the  machine's  reserved  registers  and  core 
locations. 

c.  These  instructions  which  reference  the  storage 
protection  mechanism,  the  memory  system*  or 
anything  else  that  is  specifically  used  tv  the  VMM 

h;;-Minq  and  managing  the  virtual  macnine. 

d.  Any  I/O  instruction. 


Application  of  these  hardware  rules  to  a  nuiroer  cf  familiar 
computer  systems  is  treated  in  Appendix  8.  The  results  of  these 
case  studies  are  summarized  in  the  table  of  Figure  3-?.  Except 
for  the  IBM  360  family  of  machines*  most  other  ccrtemporary  III 
generation  systems  are  rot  virtua  I ; zab  le. 
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Machine 

Hardware  Rule 
Violations 

IBM  360/67 

none 

IBM  360/65 

none 

HITAC  8400 

none 

IBM  360/85 

none 

DGC  NOVA 

none 

DDP-516 

3, 3a,  3b 

GE  635 

3a,  3c 

GE  655 

3a 

Multidata  A 

3 

XDS  940 

1 

PDP-10 

3a 

BBN  TENEX 

3a 

Note:  In  some  case,  a  machine  which  violates  hardware 
virtualization  rules  may  still  be  able  to  support  an 
HVM.  [See  Section  3.3.] 
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^GENERATION  VIRTUAL  MACHINE  CASE  STUDIES-SUMMARY 

FIGURE  3-2 
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3.3  HYBRID  VIRTUAL  MACHINES  AND  REQUIREMENTS 


As  we  have  illustrated  in  Figure  3-2  tal so  Appendix  B]  most 
III  generation  machines  .e  not  vir tua I i 23b  I e  because  of 
violations  of  the  empirical  rules.  Thus,  it  is  desirable  to 
define  another  construct  which  has  the  advantages  of  virtual 
machines.  e.g.  functional  aouivalence  to  real  machines,  but  is 
feasible  on  a  larger  class  of  systems  then  fc on. ant iona I)  virtual 
machines  are.  Furthermore  this  construct  must  be  accomplished 
without  the  disadvantages  of  the  complete  software  interpreter 
machine.  The  hybric  virtual  machine.  HVM,  defined  in  Chapter  2 
meets  both  of  these  objectives. 

Ar.  hvm  is  functional  ly  eauivalent  to  a  re3l  machine.  All 
instructions  issuec  within  the  most  privileged  layer  of  the  HVM 
ire  software  interpreted  while  all  non-or ivi I egea-l ayer 
instructions  execute  cirectly.  This  leads  to  an  ime I ementation 
on  III  generation  syste  .>  in  which  virtual  croblem  state  is 
mapped  into  physical  problem  state  but.  unlike  the  conventional 
III  generation  virtual  machine,  the  HVM  virtual  suoervisor  state 
is  rot  also  mapoed  into  ohysical  problem  state.  See  figure  3-3. 

Without  mode-maocing  and  direct  execution  of  the 
non-sensitive  supervisor  state  code  (as  in  the  conventional  III 
generation  VCS),  the  HVh  streamlines  the  empirical  hardware 
recuirements.  No  longer  need  ore  be  concerned  that  all 
non-pri vi I eged  instructions  are  insensitive  to  t*e  mode  map. 
Thus,  the  WVM  eliminates  empirical  Pule  1  from  its  set  of 
hardware  recuirements. 
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Attarots  by  the  virtual  machine  to  enter  supervisor  state 
are  directed  to  the  VhM.  Since  the  Vhh  maintains  the  virtual 
supervisor  state  as  cure  software  construct,  a  weaker  term  of 
■?ule  3a  is  all  that  is  neeced  for  WVK.  In  particular,  we  must 
identify  only  those  instructions  which  enter  supervisor  state. 
Thus,  since  supervisor  state  is  not  mapped,  a  node  cuery  need  rot 
oe  either  ard  the  instruction  is  not  sensitive.  Finally,  since 
execution  in  virtual  supervisor  state  is  under  the  control  of  an 
instruction-by-instruction  interpreter,  the  return  to  problem 
state  instruction  is  not  sensitive.  The  interoreter  itself  can 
resc  the  instruction  anc  observe  that  a  virtual  state  change  has 
occurred. 

As  i I lustratec  in  Figure  3-2  talsc  Appendix  3],  a  number  of 
III  generation  machines  fail  the  virtual  machine  empirical 
hardware  reouirements  either  because  of  Rule  1  cr  Rule  3a.  Thus, 
they  are  candidates  for  an  FVM. 
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3,4  AC  HOC  HARDWARE  "IMPROVEMENTS" 

Ip  addition  to  the  HVM,  another  approach  to  constructing 
virtual  machines  on  third  generation  architectures  micht  involve 
proposing  some  ad  hoc  "i  rrorovemer ts"  to  permit  more  systems  to 
satisfy  the  empirical  requirements  (of  Section  3,2).  We  term 
these  improvements  "ac  hoc"  since  they  respond  to  specific 
hardware  deficiencies  which  arise  in  the  use  of  the  conventional 
III  generation  software  virtual  machine  technioues.  Orly  in 
Chapter  4  do  we  consider  more  tra]or  architectural  changes  that 
follow  from  a  differert  view  of  virtual  machines. 

The  ad  hoc  imorovements  we  exo I  one  are  addressed 
predominantly  to  machines  which  violate  empirical  rule  1  or  (one 
or  more  suboarts  of)  empirical  rule  3  [Section  3.2).  We  will 
assume  that  the  machines  discusseo  in  this  section  do  not  trap  so 
as  to  cause  unrecoverable  errors.  Thus,  for  example,  we  will  not 
examine  the  OOP-516  which  traps  an  entire  instruction  late 
[Appendix  B) .  Rather,  the  difficulties  we  consider  are  either 
machines  ir  which  (1)  Senritive  instructions  c  not  trap  when 
necessary,  or  (2)  Undesirable  side  effects  are  caused  in  order  to 
guarantee  some  traD. 

Ar  examDle  of  difficulty  1  is  a  machine  in  which  some 
sensitive  instruction  is  not  privileged  and  so  Goes  net  trap  when 
sxecution  is  attempted  ir.  problem  state.  The  POP-iO  JRSTF 
instruction  is  such  an  example  (Appendix  8).  Another  case  of 
difficulty  1  is  a  machine  in  which  privileged  instructions 
execute  as  a  NOP  in  problem  state.  The  Multiaata  Model  A  or  GE 
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535  demonstrate  this  difficulty  {Appendix  R]* 

An  example  of  difficulty  ?  is  a  machine  in  which  many 
"inrocent  bystander"  core  locations  must  be  protected  in  order  to 
be  able  to  protect  some  other  sensitive  location  because  the 
machine's  relocation  or  protection  mechanism  is  too  coarse  or  too 
inflexible.  The  36G/65  or  t-TTAC  84C0  must  protect  an  entire  2K 
oyte  core  blocK  just  in  order  to  make  several  hundred  sensitive 
locations  inaccessible  [Appendix  81. 

5.4.1  Difficulty  One—  Non-trapping  Sensitive  Instructions 

Difficulty  one  may  be  restated  as  a  Droblemt  Chcose  the  set 
of  privileged  instructions  so  as  to  inc  luce  all  sensitive 
instructions  which  may  not  be  "protected"  by  some  other 
mechanism,  e.g.  protection  or  relocation  system.  The  solutior  to 
this  problem  must  be  formulated  in  such  a  way  that  if,  during  the 
course  of  VMM  design,  it  is  discovered  that  an  unanticipated 
instruction  is  sensitive,  then  the  architecture  will  still  be 
virtual izab  le. 

A  valid  solution  is  to  permit  the  supervisor  tc  make  any 
instruction  privileged  by  executing  a  special  Trap  £n  Ppcode 
(TOO)  instruction.  We  assume  that  initially  there  are  no 
orivileqed  instructions  and  that  the  VMM  is  in  ccntrcl.  The  VMM 
initializes  the  system  by  executing  a  TOO  instruction,  tor  each 
opcode,  including  TCO,  that  is  to  be  made  privileged.  At  the 
time  of  machine  design,  there  is  no  need  to  make  3  commitment  on 
which  instructions  are  to  be  privileged.  When  the  VMM  h3S  been 
written  (and  debugged)  and  it  is  known  which  instructions  are 
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It 

is  necessary  to  show 

that  the  TOO 

i nstruc  ti on 

has 

the 

lesired 

effect  and  that, 

furthermore  * 

tha  effect 

o  f 

TOO 

instructions  may  be  simulated 

for  virtual 

machines . 

For 

the 

ourposes 

of  this  discussion, 

v;e  car  assume 

that  an  occode  is 

the 

ODerand  of  a  TOO  Instruction. 

Thus*  there  are  two  new  instructions  irtroouced  into  the 


:omputer  * 


TOO —  describel  above 


UNTCC —  which  turns  off  traoping  for  the  operand  opcode 
and  sets  a  condition  coce  indicating  whether  trapping 
was  previously  on. 


figure  3-4  gives  a  simplified  example  (of  one  possible 
implementation)  in  cornectior  with  the  IFM  350.  The  operand 
field  is  assumed  to  be  ir  bits  long.  The  TRAP  register  is  2**m 
bits.  IThe  trap  register  is  shown  as  a  bit  mask.  This  is  Just 
one  possible  implementation.  For  example*  it  could  be  an 
associative  memory,]  Execution  of  the  instruction  "TCO  o"*  where 
o  is  a  valid  opcode*  causes  bit  p  of  the  TPAP  mas*  to  be  set  to 
1.  Subseauent  execution  (ir  orotlem  state)  of  any  instruction 
whose  bit  is  turned  on  in  the  trap  register  causes  a  trap  to 
occur.  Thus*  sensitive  instructions  may  be  made  orivileged. 

The  action  of  the  TOO  and  UNTOO  instructions  may  be 
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TRAP  MASK  REGISTER 


TOO 

TOO 

TOO 

UNTOO 

TOO 

SSK 

TOO 

ISK 

TOO 

SVK 

TOO 

SSM 

TOO 

LPSW 

Initialization  sequence  for  modified 
360  with  trap  register. -Status 
before  'TOO  LPSW'  instruction 
is  executed. 

Instruction  is  privileged  if  bit  set 
in  trap  register. 


TRAP  REGISTER  AND  TOO  INSTRUCTION 
FIGURE  3-4 
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simulated  for  virtual  machines.  The  VMM  keeps  a  codv  of  the 
virtual  machine's  trac  register,  i.e.  virtual  trap  register, 
attempted  execution,  by  the  virtual  machine  (running  in  problem 
state,  as  before)  of  the  TOO  or  UNTOO  instruction  causes  a  trap 
to  the  VMM.  The  supervisor  then  simulates  the  effect  or.  the 
virtual  trap  register.  Before  the  virtual  machine  is  dispatched 
in  (virtual)  problem  state,  the  virtual  trap  register  is  OR'ed 
into  the  (real)  trap  register.  Figure  3-5  illustrates  hew  the 
virtual  trap  register  is  supported. 

3.4.2  Difficulty  Two —  Undesirable  Protection  Sice-effects 

The  problem  considered  here  is  largely  one  of  inelegance, 
awkwardness,  or  unreasonable  performance  degradation  caused  by 
the  side  effects  of  mechanisms  reouired  in  order  to  limit  access 
to  certain  sensitive  (core)  locations.  Most  computer  systems 
have  a  rumber  of  sensitive  locations,  often  termed  the  interrupt 
control  area  1691.  Since  the  interrupt  control  area  is  located 
in  physical  core,  it  may  be  tampered  with  by  non-or- i  v  i  I  eged 
instructions.  Consecuen t I y ,  it  is  necessary  (3S  brought  out  in 
Section  3.2)  to  protect  these  pnysical  locations  by  means  of  the 
orotection  or  relocation  hardware  of  the  host.  Unfortunately, 
the  protection  or  relocation  system  is  often  too  "coarse"  to 
oerform  this  task  neatly  and,  as  a  result,  other  nen-sensi t i ve 
locations  may  be  protected  as  well. 

There  are  roughly  two  poirts  of  view  that  Tav  be  adopted  to 
avoid  cifficulty  two.  In  the  first  solution,  we  alter  the 
machine  design  to  provide  a  finer  grain  protection  mechanism. 
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TRvm  —  VM  trap  register 

TRVmm —  VMM  trap  register 

TR$  —  traps  for  sensitive  instructions 

to  run  virtual  machine - 

TRvmm  •  "  TRvfn  V  TRS 
trap  on  opcode  p  — 


VIRTUAL  MACHINE  SUPPORT  FOR  TRAP  REGISTER 


FIGURE  o-5 
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Instead  of  protecting  core  or  the  basis  of  2K  clocks  (as  in  the 
160/65)*  we  can  do  it  or  a  word  ba;is.  We  introduce  3  scecial 
new  irstruction  Trap  On  Locatior,  TCL,  whicn  may  be  used 
analogously  with  TOC  (above)  to  initialize  the  sensitive 
locatiors  for  a  virtual  machine.  After  a  location  has  been 
TOLed*  an  attempt  to  reference  it  ir  oroblem  state  causes  a  trap. 
There  are  a  number  of  different  ways  in  wrich  TOL  may  be 
implemented.  For  example*  an  associative  memory  car.  be  used  to 
hold  locatiors  that  have  been  TOLed.  Or*  each  word  of  memory  may 
be  associated  with  one  bit  which  inaicates  if  it  causes  a  trap  on 
reference.  This  second  i me  lementation  has  beer  used  in  the 
design  of  the  ISPL  machine  at  the  SAND  Corporation  111,12).  The 
first  imp  lementatior  with  a  one  word  (associative)  memory  has 
been  used  in  the  IBM  M44  (91)  and  the  MIT  Art  ficial  Intelligence 
Laboratory  FOP-10  161,62)  to  simulate  aedress  stoc  switches. 

In  the  secono  solution*  we  remove  the  sensitive  locations 
from  main  memory  and  place  them  ir  some  uraddressab  le  or 
differently  addressable  memory.  In  order  to  access  this  special 
memory,  we  reauire  a  rew  privileged  instruction.  See  Figure  3-6. 
In  some  machines*  e.g.  360/67*  37C*  this  type  of  auxiliary  memory 
is  sometimes  called  control  memory  or  control  registers.  There 
are  several  good  reasons  for  acooting  this  solution.  First 
associating  sensitive  locations  with  privileged  instructions 
which  access  them  provides  a  homogeneous  mechanism  for  virtual 
machine  trapping.  Another  reason  for  adopting  solution  two  is 
that  some  of  the  locations  in  the  interrupt  control  area  are  an 
intimate  part  of  the  description  of  the  state  of  the  processor. 


A 

% 

-V 


*Kt/ 


Page  65 


MEMORY 

ADDRESSABLE 

MEMORY 


SENSITIVE  LOCATIONS  IN  UN  ADDRESSABLE  MEMORY 


FIGURE  3-6 


iZa%? * »  >  » -  -.'  iST-JIA/i 


Page  f6 


Thus*  thev  should  be  associated  with  the  processor  *on  a  more 
individual  basis)  as  the  PSW  is. 

Solution  two  as  an  ad  hoc  improvement  was  recognized  by 
Puchi  et  al  Z5G]  in  their  worw  with  a  virtual  machine  for  the 
HTTflC  PAD). 


We  have  two  proposals  for  further  hardware 
improvmepts.  One  is  that  memory  orotection  should 

become  more  flexible  to  irclij*  reed  orotection. 
The  other  is  that  the  CAW  stor^e  area  and  the 
timer  shou I c  be  moved  from  the  main  core  memory  to 
the  scratchpad  memory.  The  latter  takes  some 
trouble  in  simulator  construction  3nd  is  inelegant 
from  the  logical  designer's  point  of  view  C 5 C 3 . 

If  all  of  these  reouirements  are  met*  we  coull 
bui I o  a  more  natural  Simula  for  ...  [501. 


As  mertionecJ  above,  the  System/370  incoroorates  some  aspects 
if  solution  two.  Some  of  the  new,  sensitive  locatiors  nave  been 
made  "control  registers"  and  are  accessible  only  via  privileged 
instructions.  The  new  channel  and  CPU  identification  numbers  are 
invisible  and  accessible  only  via  privileged  instructions.  The 
new  37:  clock  is  invisible  and  can  be  stored  into  only  with  a 

orivileqed  instruction.  Unfortunately*  it  may  be  reac  by  a 
non-orivi I eged  instruction. 
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3.5  Til  GCNfo/sriON  EMPIRICAL  SOFTWARE  REO'JlREf'EKTS 


Tn  Section  3.2,  we  developed  the  empirical  hardware 
~eouirements  for  lyoe  I  virtual  machines.  In  this  secticn»  we 
add  the  empirical  software  requirements  that  must  he  introduced 
for  Tyce  II  (extended  machine  host)  virtual  machines.  Recall 
that  a  Tyoe  II  virtual  machine  system  [Chapter  21  is  one  in  which 
the  virtual  machine  monitor  runs  under  a  host  operating  system 
rather  than  on  a  bare  hardware  host  machine. 

Thus  in  Figure  3-*7,  hardware  events  such  3s  the  trap  caused 
oy  an  attempt  to  execute  a  privileged  instruction  in  the 
supervisor  state  of  the  virtual  machine  tohysiC3l  nrotlem  state!, 
get  passed  to  the  hcst  operating  system.  This  operating  system 
then  gives  control  to  the  VMM  (under  it)  which  may  perform 
privileged  instruction  simulation  for  the  virtual  machine. 

A  Type  II  virtual  machine  may  be  desirable  in  a  number  of 
c ircumstances  as  pointec  out  in  Chacter  ?.  Because  a  Type  TI 
virtual  machine  supervisor  has  a  (possibly)  rico  set  of 
supervisory  services  (SVC’s)  available  to  it  courtesy  of  the  host 
operating  system,  it  often  requires  less  effort  to  inclement  a 
Type  II  system  than  a  Tyoe  I  system.  At  the  s.me  time,  however, 
the  SVC's  and  the  SVC  mechanism  must  be  of  a  form  that  does  rot 
make  it  impossible  to  implement  a  Type  II  virtual  machine. 

Thus  the  dilemma.  The  host  ooerating  system  must  provide 
those  supervisory  services  (primitives)  that  mass  efficient  ‘ne 
implementation  of  a  Tyoe  II  virtual  computer  system,  without 
making  it  imoossiole. 
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Since  the  empirical  software  reauirements  Mill  comolement 
the  empirical  hare-ware  r  eauiremerts  ,  we  begin  our  search  with 
another  look  at  the  requirements  of  Section  3.2.  Empirical 
hardware  rule  i  still  remains  valid.  The  introduction  of  the 
additional  structure  *  an  operating  system  between  the  virtual 
machine*  VMM,  and  the  hardware  must  do  nothing  to  invalidate  rule 
1. 

The  Tyc«  II  system  version  of  rule  2  is  th3t  there  must  be  a 
oriiritive  available  to  protect  the  virtual  machine  monitor  from 
the  active  virtual  machine.  Since  tne  virtual  machine  monitor 
will  be  running  ir  ar  insulated  environment  that,  likely,  will 
prevent  it  from  utilizing  +he  crotection  or  relocation  hardware 
directly,  there  must  be  a  system  primitive  that  it  can  invoke  to 
obtain  the  desired  effect. 

Pule  3  must  be  modified  to  indicate  that  when  the  virtual 
machine  traps,  the  ooerating  system  supervisor  directs  the  signal 
to  the  VMM  for  processim.  Since  the  operating  system  ard  VMM 
are  no  longer  a  single  unit,  there  must  be  a  method  of 
com.municat ire,  to  the  ooerating  system  supervisor  that  the  virtual 
machine  is  a  oarticular  subprocess  of  the  VMM  ar.d  that  all  cf  the 
appropriate  virtual  machine  traps  are  to  be  directed  to  the  VMM. 
The  mechanism  for  setting  up  the  sufcprocess  and  indicating  where 
the  signal  is  to  go  may  be  accomplished  either  bv  a  formal 
sufcprocess  primitive,  or  it  may  be  an  informal  crocedure  that  is 
mace  up  of  a  number  of  the  other  primitives.  It  is  not 
sufficient  for  the  virtual  machine  to  mere’y  signal  the  VMM.  As 
before,  the  trap  from  virtual  machine  to  VMM  (now,  by  way  of  the 
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jperatinj  system)  must  not  cause  unrecoverable  errors.  It  must 
be  possible  for  the  VMM  to  simulate  the  effect  of  the  ooeration. 
Thus,  it  is  not  satisfactory  to  set  uo  a  subcrocess  in  which 
anything  that  apoesrs  anomalous  is  considered  to  be  an  error  and 
the  subprocess  is  destroyed.  This  is  actually  fairly  common 
among  systems. 

Since  primitives  (SVC*s)  alter  or  query  the  state  of  the 
machine,  they  are  sensitive  according  tc  hardware  requirement  3a. 
Thus.  their  attempted  execution  by  the  virtual  machine  is 
forbidden  and  must  cause  a  trap  to  the  virtual  machine 
supervisor.  Conseouent I y .  the  operating  system  requires  a 
oriiritive  that  may  be  used  to  turn  on  or  off  the  SVC*s  for  a 
subcrocess.  In  the  case  of  a  virtual  machine,  the  VM*  would  use 
this  primitive  to  shut-off  all  SVC's.  ~  .is.  of  course,  includes 
the  SVC*s  to  turn  on  and  off  the  SVC*s. 

Note  that  in  a  well  desigred  ’user  machine*  142.751.  system 
primitives  replace  the  hardware  instructions  to  accomplish  the 
sersitive  instructions  a.b.c.  and  d  of  Section  3.2.  Thus, 
shutting  off  the  SVC*s  makes  these  sensitive  instructions  (along 
with  all  other  SVC*s)  trap  in  the  *user  machine.* 

Thus,  we  are  able  to  state  the  following  empirical  software 
<-ules  which  govern  whether  a  Type  II  (extended  mschire  host)  VCS 
may  be  implemented  under  the  operating  system  of  a  virtual  izab  le 
third  generation  computer  system. 
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(SI)  The  irethod  of  instruction  execution  of 
non-privi legec  instructions  in  both  supervisor  and 
problem  state  must  remain  equivalent  for  a  large  subset 
of  the  instruction  repertoire. 

(S?)  A  primitive  for  protecting  toe  virtual  machine 
monitor  (and  any  other  virtual  machine  if  there  is 
multiprogramming  under  this  VMM)  from  the  active 
virtual  machine  must  be  available.  This  may  be 
accomplished*  for  example*  through  a  orotection 
primitive*  address  translation  primitive*  or  some  more 
general  subprocess  primitive. 

(S3)  A  primitive  to  tell  the  operating  system 
supervisor  to  signal  the  virtual  machine  monitor  when 
the  virtual  machine  attempts  to  execute  a  sensitive 
instruction  must  be  available.  The  signal  to  the  VMM 
must  not  cause  unrecoverab I e  errors.  It  must  then  be 
possible  for  the  virtual  machine  monitor  to  simulate 
the  effect  of  the  instruction.  e  sensitive 
instructions  are  the  same  as  for  the  hardware 
requirements. 


Aoolication  of  these  software  rules  to  several  computer  systems 
is  treated  in  Aopencix  C.  The  results  of  these  case  studies  are 
summarized  in  Figure  3-8.  Except  for  the  ITS*  Incompatible 
Timesharing  System,  which  orovided  the  necessary  crimitives*  the 
operating  systems  examined  either  failed  the  empirical  rules  or 
reauired  ad  hoc  or  specialized  solutions. 
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Operating  System  Software  Rule  Difficulties 

HITAC  8400  TOS/TDOS  Ad  hoc  solution  (Se+  S3) 
IBM  360/67  UMMPS  Specialized  solution  (So+S 


IBM  360  0S/360 
DEC  PDP-10  ITS 
IBM  360  CMS 


Specialized  solution  (S2+S3) 
(SVC  SWPTRA) _ 

Violates  rules  (S2+S3) 

Satisfies  rules 

Specialized  solutions  are 
possible. 


HI  GENERATION  TYPE  E  CASE  STUDIES -SUMMARY 

FIGURE  3-8 
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3.6  SOFTWARE  rm  .MITIVFS  FOP  TYPE  IT  VCS'S 


As  we  have  illustrated  in  Figure  3-8  tal  so  Acpendix  Cl*  frost 
III  generation  ooersting  systems  are  not  designed  to  suooort  Type 
II  VCS's.  The  systems  whicn  do  have  largely  beer  reworded  to  do 
so.  From  the  experience  with  the  ITS  Type  II  HVM  [Appendix  Cl  it 
becomes  possible  to  indicate  the  Kind  of  operating  system 
oriiritives  that  would  simplify  the  support  of  Type  II  third 
generation  virtual  machines.  To  avoid  the  usual  problems  with 
asynchronous  I/O.  we  will  discuss  the  primitives  only  with 
respect  to  the  CPU  and  memory  of  the  virtual  machine. 

The  ITS  UUO's.  for  example,  provide  facilities  slightly 
different  from  those  needed  in  supporting  virtual  machines. 
Since  the  goal  of  the  ITS  primitives  is  to  suooort  a''  extenoed 
machine  environment,  the  primitives  provide  greater  flexibility 
thar  is  needed  for  virtual  machines.  For  example,  in  ITS  a 
superior  may  destroy  all  of  the  inferiors  in  the  sub-tree  below 
it.  In  a  virtual  machine  environment,  the  VMM  need  only  be  able 
to  destroy  an  immediate  inferior,  since  that  is  the  extent  cf  the 
sub-tree.  SuDport  of  additional  levels  of  virtual  machines  is 
transparent  to  the  VMM  and  resides  with  the  second  c.>y  cf  ITS 
and  the  second  VMM  which  are  running  in  the  level  one  virtual 
machine.  The  copy  of  ITS  in  the  reel  machine  does  not  Know  about 
the  level  two  virtual  machine  and  only  sees  a  level  one  machine. 
Hence,  the  orimitive  is  more  general  than  is  specifically  needed 
for  VMM  construction.  See  Figure  3-9. 

Other  primitives,  such  as  the  Cennis-Vsn  Horn  orimitives 
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VMM 


I.I.2.2 


VM 


Typical  Multi- level 
process  tree 


Type  I  Two-level 
Virtual  Machine  Tree. 

The  VMM  only  sees  one 
level  beneath  it. 


PROCESS  TREES 
FIGURE  3-9 


‘-  **iifp«fit% a5*  ^*fcl<M>»5VJ*’M &Ab>  - 


Page  76 


virtual  machire  i  and  write  into  word  w  of  the  VMM. 

(A)  set  Status  i«)*wT 

Write  corterts  of  VMM  word  w  into  virtual  register 
or  memory  location  |  in  virtual  machine  i. 
f*>  fieslCfi*  it 

Destroy  virtual  machine  i.  This  ceririts  the 
identifier  i  to  be  used  again. 

Given  that  the  host  machine  satisfies  the  Type  I  hardware 
r eauireirents*  then  a  Type  IT  VMM  (non-HVM)  may  be  crogrammed 
jsing  these  -rimitives. 


vmmi  i l  =  vjrty? I  machine  loc*  R56K,  ...  t 
4iSI2.2l££  it 

locJ  igt£h  status  i»l»w* 

(SIMULATE  OFFENDING  INSTRUCTION) 

SSl  slatyS  i,J,w: 
diSEslih  it 

SDJ? 

The  amount  of  status  fetching  and  setting  and  simulation  of 
sensitive  inc.tr  .c  t  ions  varies  according  to  the  machine’s 
complexity  and  the  difficulty  of  virtualizing.  .'n  a  III 
■jcreraticr  machine  which  is  easily  vi  rtua  I  i  zab  I  e»  the 
implementation  of  the  VMM  sketched  above  can  be  reduced  to  a 
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3.7  CONCLLSION 


The  main  result  of  Chapter  3  has  been  the  substantiation  of 
the  unsuitability  of  most  existing  ITT  generation  systems  for 
supporting  virtual  machines.  We  have  shown  that  the 
architectural  characteristics  of  HI  generation  systems  and  the 
definition  of  virtual  machine  leaC  to  a  particular  software 
construction  of  the  virtual  machine  mao.  For  example,  the 
processor  component  of  this  software  map  takes  the  orivileped 
layer  of  the  virtual  processor  into  the  unprivileged  layer  of  the 
real  machine.  This  immediately  leads  to  3  set  of  emoirical 
reouireirentst  e.g.  instruction  execution  must  be  insensitive  to 
the  software  layer  maoping.  which  distinguish  III  generation 
virtua I  izab  le  architectures .  I  number  of  examples  reveal  the 
startling  fact  that  most  third  generation  architectures  are  not 
virtua I izab  le.  That  CP-67.  the  best  known  virtual  machine 
system,  could  be  constructed  on  the  IBM  361/67  was  a  happy 
accident . 

In  response  to  difficulties  with  virtualizing  most  TTI 
generation  architectures  we  propose  two  alternatives.  First,  we 
introduce  the  hybrid  virtual  machine  which  has  less  stringent 
hardware  requirements  but  potentially  more  overhead  than  the 
virtual  machine.  Then,  we  propose  a  number  of  simple  ad  hoc 
changes  that  might  be  made  to  III  generation  machines  to  simplify 
virtual izat ion. 

The  rest  of  Chapter  3  is  concerned  with  implementing  a  Type 
II  (extended  machine  host)  VMM  on  a  third  generation  operating 
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system.  The  virtual  machine  construction  is  basically  the  same 
as  before.  However ,  now  the  VMM  must  manipulate  its  components 
using  the  orimitives  of  the  host  operating  system.  This  leads  to 
3  series  of  empirical  requirements  *  e.g.  there  must  be  an 
operating  system  primitive  to  shut  off  all  SVC*s  of  the  virtual 
machine.  that  distinguish  Type  II  virt ua I i tab  I e  operating 
systems.  Cr.ce,  again,  most  third  generation  operating  systems 
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However,  the  main  point  of  Chapter  3  has  been  that  no  III 
generation  systems  fha'cware  or  software)  have  been  designed  with 
the  intention  of  supoorting  virtual  machines.  This  has  forced 
virtual  machines  to  be  introduced  via  a  particular  III  generation 
software  construction.  Requirements  imposed  by  this  construction 


in  turn,  have  disqualified  almost  all  III  generation  machires 
from  supoorting  virtual  machines<-  Even  the  few  existing  systems, 
e.g.,  CP-67,  require  considerable  software  support  which  implies 
development  and  implementation  costs,  and  sub-oct imal I y  efficient 
operation. 

These  consequences  lead  us  to  search  for  machine 
architectures,  specifically  intendeo  to  support  VCS*s.  As  noted 
in  Chaoter  1,  these  considerations  gave  rise  tc  the  crocosals  by 
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Lauer  and  Snow  C7f;}  and  Gagliardi  ard  Goldberg  Col].  While  the 
ticdel  of  Ch3oter  4  is  developed  specifically  in  connection  with 
providing  for  the  orderly  introduction  of  virtual  machines  into 
the  IV  generation*  the  principles  which  result  are  3*ner':,l,> 
applicable  to  all  archi tectures.  Thus*  by  suitable 
interpretation  of  the  model  we  can  develop  orincioles  for  III 
jeneration  systems.  This  approach  is  illustrated  in  Sectior  4.9. 
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!  CHAPTER  U . 

PRINCIPLES  FOR  IV  SENCRAT ION  VIRTUAL  COMPUTER  SYSTEMS 
The  Virtual  Machine  Mao  and  its  Firmware  Imo I ementst ior 
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<«  •  G  PLAN  OF  CHAPTER 


In  Chapter  4»  we  will  discuss  the  problem  of  introducing 
virtual  computer  system  capabilities  into  the  IV  generation.  A 
~ecent  paper  151]  points  out  some  of  the  inherent  difficulties 
and  complexities  caused  by  likely  IV  generation  architecture  and 
argues  for  a  hardware-f ir mware  assisted  virtual  izer  built  into  IV 
generation  computer  systems.  In  this  chapter*  we  carry  the 
notion  of  hardware  virtualization  somewhat  furtner  and  provide  a 
general  framework  for  viewing  virtua I izat i on  of  a  IV  generation 
computer  system.  Furthermore*  the  resul t  that  eierges  is 
applicable  to  less  complex  archi tectures  as  well. 

Section  A.l  introduces  the  likely  char.act  er  is  t  i  cs  of  IV 
generation  computer  systems  architectur e.  The  orircical 
archi tectura I  development  of  such  systems  will  be  a  formalized 
firmware  implementation  of  processes.  A  corresponding  model  for 
processes  is  introduced*  the  process  map*  4*  maos  process  names 
into  resource  names. 

Section  A. 2  develops  a  model  for  a  IV  generation  VCS.  He 
introduce  the  virtual  machine  mao  (or  resource  mac)  f  which  maps 
virtual  resource  names  into  real  resource  names.  Then  the 
virtual  machine  map  is  combined  with  the  process  mac  of  Section 
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>♦.1.  Tne  composition  of  the  two  maopings*  f  c  fc,  expresses  the 
-napping  from  process  names  to  real  resource  names*  and  thus 
expresses  the  maoDing  of  names  accessible  to  a  process  executing 
on  a  virtual  machine*  Other  related  notions*  such  as  recursion 
and  Type  IT  virtual  machines  are  introduced  into  the  model.  Some 
elementary  properties  of  tne  maps  and  virtual  systems  sre 
enumerated. 

Section  4.3  uses  the  model  of  Section  4.2  to  point  out  the 
unsuitability  of  £.  software  implementation  of  the  msp  f  fas  was 
studied  in  Chapter  3)  for  a  IV  generation  VCS.  The  argument 
depends  on  the  complexity  of  the  process  mop*  4,  and  the 
likelihood  of  4  visibility  by*  say*  tne  inner  layer  supervisory 
software.  ft  specific  example  of  the  kind  of  difficulty  3 
software  i rp lementat ion  of  f  might  cause  is  developec  for 
protection  rings. 

Section  4.4  discusses  the  Venice  Proposal,  V3,  introduced  in 
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"Virtua I izeat I e  Architectures"  1511.  The  VP  Drovioes  an  interim 
proposal  for  the  design  of  a  hardware-firmware  assisted 
virtual izer  (HV)  for  a  IV  generation  VCS.  The  advantages  and 
disadvantages  of  the  VP  are  pointed  out  using  the  model  of 
Section  4.2. 

Section  4.S  introduces  the  design  of  a  hsrdware-f i rmware 
virtualizer  based  directly  upon  the  model  cf  Section  4.2.  The 
initial  objective  is  automatic  virtualization  of  CPU-Kemory 
processes  (with  a  minimum  of  software  interventi on) .  The  new 
instructions  introduced*  modifications  to  the  system  base  and 
expected  cerformance  are  all  indicated. 
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Section  4.6  provides  a  number  of  detailed  examples  to 
illustrate  instruction  executions  on  virtual  computer  systems 
constructed  with  representst ive  hardware  v irt ua I i zers. 

Section  4.7  briefly  considers  some  of  the  subtler  problems 


raised  earlier  in  the  chapter.  Proper  virtual  treatment  of 


i nput/output,  a  technologies!  difficulty,  must  await  progress  in 
the  real  treatment  of  inout/outout .  In  the  interim,  some  sd  hoc 
solutiors  to  I/O,  derived  from  III  generation  techniques,  are 
provided. 

Section  4.3  applies  tne  model  of  Cnaptar  4  to  3 
reconsideration  of  second  and  third  generation  computer  systems. 
One  design  that  emerges,  the  **f=R-B,  ^identity"  machine,  is 
similar  to  the  machine  recently  proposed  by  Lauar  and  Snow  [763. 
The  other  design  yields  a  modified  IBM  35;  with  a  greatly 


stream  I ined  CP-67 


u.l  CHARACTERISTICS  OF  IV  GFNERATION  COMPUTER  SYSTEMS 


The  mosl  distinctive  architectural  characteristic  that  will 
be  found  ir  IV  generation  machires  Hill  be  the  refinement  and 
formal  imo  I  ementat i on  in  firmware  cf  the  process  model 
[31,37, Al],  According  to  this  view*  activities  in  a  computing 
system  Mill  a  performed  by  a  number  of  cooperating  seauential 
processes,  each  accomplishing  some  elementary  task  and 
communicating  among  themselves  via  a  well-defined  mechanism* 
Fur thermore ,  the  processes  themselves  will  execute  in  an 
environment  extended  to  include  idealized  objects*  such  as  an 
idealized  address  space. 

This  view  of  the  likely  characteristics  of  IV  generation 
architecture  is  based  on  the  papers  by  G.M.  and  L.Q.  Amdahl 
15,7],  the  Plauuw  peoer  I191*  the  *orris  P3Der  The  Infotech 
oook  [70],  and,  particularly,  on  the  existing  Mitre  CcrDor3tion 
VENUS  system  as  ctascrioed  by  Liskov  [791.  Furthermore,  we 

observe  that  key  software  features  of  generation  o  are  often 
those  features  that  are  included  in  the  interior  decor  of 
generation  cii*  Therefore,  Denning’s  recent  tutorial  1 37  3  on 
(the  software  of)  Late  Third  Generation  systems  strongly  suggests 
that  the  process  model  will  be  incorporated  into  future  systems. 

Third  generation  computer  systems  rely  on  3n  ad  hoc,  purely 
software  imo lementstion  of  processes  [37].  Tnus,  tre  choice  of 
primitives,  organization  of  tables,  names  of  oojacts,  etc.,  is 
not  fixed  3nd  may  be  chosen  by  the  software  sucoort.  This 
implies  th3t,  for  a  given  III  generation  system,  there  exists  no 
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jnifiua  imp  I  enen  tat  ion  of  processes. 

By  contrast,  IV  generation  comoute:  systems  hi  I  i  feature  a 
firmware  imol ement3tion  of  processes.  The  complete  collection  of 
status  information  for  all  processes  in  the  syste-n  will  ae  called 
the  system  base  (51,871.  The  firmware  will  Know  the  base 
location  of  the  system  base*  the  SCOT.  Since  tables  and  control 
oiocks  will  be  of  fixed  (ore-assigned)  format,  kncwledqa  of  the 
300T  is  sufficient  for  the  firmware  to  locate  any  element  in  the 
system  base.  The  system  base  will  be  located  in  main  memory. 
For  largely  economic  reasons  151,07],  the  system  base  will  likely 
a  I s  c  be  accessible  to  the  inner  layer,  roos*  privileged  software 
Csee  below].  However,  during  process  execution,  the  system  base 
is  referenced  and  updated  directly  by  the  firmware. 

The  system  base  will  incluce  elaborate  orocess  control 
blocks,  aueueing  data  bases,  address  space  words,  etc.  (51,87]. 
See  Figure  4-1.  Tne  queueing  csta  bases  will  ce  usee,  for 
example,  by  tne  firmware  dispatcher  to  invoke  the  READY  orocess 
with  nighest  priority  (991.  The  oueueirg  data  oases  will  also  be 
used  by  the  f  irmwsre  in  implementing  interprocess  communication 
via  Oilkstra's  P-operstiors,  V-operat i ons ,  and  semaphores 
(41,791 . 

The  orocess  control  blocks,  (PCB’s)  will  be  coirted  to  by  3 
orocess  table  whose  location  can  be  derived  from  the  90CT.  Thus, 
an  integer  displacement  in  the  orocess  table,  the  process-id, 
identifies  a  oarticular  process  and  tne  location  of  its 
corresponding  PC9.  The  PCH  will  contain  suen  information  as 
temporary  storage  of  the  central  processor’s  (CPU’s)  register 


values*  the  priority  of  the  process*  ana  ar  admass  space  word. 
The  CPU  will  contain  a  process-id  register  whosj  value  identifies 

the  process  in  execution.  Thus*  there  exists  no  notion  of  an 
absolute  raooe;  execution  is  always  on  behalf  of  sore  process. 
Loading  the  process-id  register  activates  the  subject  process  and 
instructs  the  firmware  to  obtain  register  values*  etc.  frem  the 
process  control  bloc*  (FC'd) .  Tn  particular,  tne  address  space 
wore  is  referenced  and  used  to  form  the  memory  m3D.  Thus,  there 
exists  no  notion  f  an  absolute  addressing  mode?  all  addresses 
are  mapped. 

The  icealized  sdcrrss  space  will  likely  be  a  segmented 
layered  structure  1 15*35*  *}£*  d2*tf'6 I  •  Such  a  structure  can  be 
viewed  as  a  generalization  of  *he  two  layer  Master/Slave  modes  of 
ITI  generation  systems.  As  the  layer  or  ring  II ‘61  numoer  of  an 
3Ctive  process  decreases*  tne  ability,  bv  that  crocess,  to  access 
segments  in  its  segment  table  increases  corre soono i no  I y .  In  ring 
0»  the  most  privileged  layer*  supervisory  processes  can  also  be 
expected  to  utilize  a  small  set  of  necessarily  privileged 
instructions.  Privileged  ring  procedures  of  processes  will  also 
likely  be  invoked  oy  process  exceptions.  This  is  the 
generalization  r <  the  trap  to  master  mode  in  III  generation 
•systems . 

The  I/O  systems  of  IV  generation  computer  systems  will 
likely  be  similar  to  those  of  the  late  III  generation*  e.g.  IBM 
System/?/'?.  In  particular*  I/O-mernory  processes  will  very  likely 
not  oe  formalized  to  the  extent  that  CPU-m  c.i  ary  processes  have 
been.  I/O  processor  (channel)  operations  will  execute  in  an 
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absolute  mode  without  process-id's  or  segmented  addresses  18]. 
All  address  absolutizing,  status  maintenance,  etc.  will  be 
oer formed  by  software  in  an  ad  hoc  manner  1701.  As  a  result  of 
the  lack  of  progress  by  IV  generation  designers  in  I/O  process 
formalization  and  implementation,  the  IV  generation  virtual  I/O 
considerations  are  largely  the  same  as  those  of  the  III 
generation.  Consequently,  most  of  the  discussion  of  Chapter  4 
will  omit  I/O?  only  in  Section  4.7  will  wa  illustrate  III 
generation  technioues  tor  introducing  virtual  I/O  in  IV 
generation  systems. 


J\bSlC2£llJ20 

A  model  of  (CPU-meaory )  process  structure  nay  be  obtained  by 
3  scrutiny  of  the  system  base.  Execution  by  the  CPU  is  always  on 
behalf  of  some  process.  Hence,  the  names  used  by  individual 
instructions  are  always  process  names,  whether  they  are  local  or 
global  (system-wide)  names.  For  example,  data  or  instruction 
addresses  are  always  local  names,  defined  by  the  segment  table  of 
the  active  process  control  block  (PCB).  Process-id’s  are  global 
orocess  names  and  their  meaning  is  defined  for  all  processes  via 
a  single  table  in  the  system  base. 

Effectively,  then,  the  system  base  relates  two  classes  of 
objects — those  used  by  processes  (whether  local  or  global)  and 
those  related  to  the  actual  resources.  The  process  names  are 
idealized  constructs  while  the  resource  names  are  cetermined  by 
the  hardware.  There  is  no  requirement  that  orocess  names  have 
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the  same  torn  or  structure  as  resource  nanes.  Thus,  in 
particular,  it  becomes  ocssible  to  sne?i«  of  5  process  having  a 
segmented,  Tu  1 1  i-di  irer  s  ions  I  address  space.  In  contrast,  the 
real  (and  virtual)  memory  resource  address  sdbcs  nust,  of  course, 
be  lirer-r  to  mirror  ohysicsl  construction.  The  distirction 
oetween  nerivea  and  orimitive  resource  names,  such  as  a  segmented 
and  Iire3r  memory,  is  discussed  in  "Vi rtua I iz *ab I  a  Archi tectures" 
(511 . 

We  call  the  names  used  by  processes,  orocess  names  or 
■’-names  and  those  use!  by  tne  hardware,  resource  names.  In 
oarticul3r,  the  real  resource  r&mes  used  by  the  ohysical  hardware 
ere  called  P-names.  [However,  as  we  shall  see  aelow,  V-names  are 
also  resource  names.]  It  will  be  necessary  to  discuss  3  manning 
function,  6,  which  is  implemented  via  the  system  base  ano  mans 
orocess  names  into  resource  names.  See  Figure  4-2.  tin  Figure 
4-2  and  all  figures  that  follow  in  Sections  4, 1-4.4,  orocess 
spaces  are  represented  by  circles  and  resource  scaces  by 
sauares. 1 

Let  the  process  space  names  be  P=Cpf>,ol,  ...,  oa).  As 
liscussed  above,  P  includes  the  process  memory  scoce  names  and 
semaphore  names  (for  the  active  process),  all  or ccess-i d * s ,  etc. 
Let  R=Cr),rl,  ...,  rb}  be  the  (real)  resource  soace  names.  R 
also  includes  the  processor  and  (at  least  c onceatual I y)  the  1/0 
resource  names,  as  well  as  the  memory  names.  Then,  for  the 
active  orocess,  we  orovide  a  way  of  associating  process  names 
with  (real /virtual )  resource  names  curing  process  execution.  See 
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$  :  P  ~>  R  U  {e} 

<j)(x)  =  (y  if  y  is  the  resource  name  for  process  name  x 
e  if  x  does  not  have  a  corresponding  resource 
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Examples  : 


<f>  ( 1  j  142)  =  6142 

if  seg  1  at  locotion  6000 


4>  (12 1  3)  =  e 

if  seg  12  does  not  exist 


<£  ( sem  2 )  =-  5000 

if  sem  2's  header  at  location  5000 


(process  id  40)  =  e 

if  process  40  does  not  exist 


THE  PROCESS  MAP  <f) 
FIGURE  4-2 
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Figure  4-2*  To  this  enc,  via  the  system  l;ase,  we  define,  for 
each  moment  of  time*  3  function 
ft:  P - ><?  U  Cel 

such  that  if  x  e  P»  y  e  P*  then 


ft(x)  =  Cv  if  y  is  the  resource  name  f cr  orocess  nsme  x 
C 

Ce  if  x  does  not  have  s  ccrr  esoond  ing  resource 
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The  value  ft(x)=e 
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procedure. 
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machine* 
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[ Section 

4.21,  proc 

Section  4.6  provides  detailed  examples  of  th»  orocess  map* 
ft.  For  clarity,  a  few  simple  examples  fellow.  Throughout 
Chapter  4,  we  adopt  the  segmented  address  rotation,  sld,  Hhere  s 
is  the  segment  number  and  d  is  the  offset  within  the  segment 
(86,923. 


Examples:  ft  (1 1 142) =6142  if  segment  1  is  located  at  6010 


ft  (T  HI,  3 1 )  =e  if  seg  2.T  does  not  exist 


ft  (3 1 5  ji'jO  )  =  e  if  5  outside  bounds  zf  segment  ^ 


ft  (sein  2)=?1G'j  if  sem  2*s  header  at  locat;on  5  1 1  r 
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4.2  f'OOEL  OF  A  TV  GENE9AT  ION  VCS 


In  order  to  prooerly  model  the  runrino  of  3  process  on  a 
virtual  mactine*  it  is  necessary  to  first  clarify  what  is  meant 
by  a  virtual  machine  and  what  are  virtual  resources.  To  do  this, 
we  formally  introduce  and  identify  an  important  njw  conceot,  the 
v  irtua  I  machi  ne  m^g  (VMirap)  or  resource  map.  The  VMmao  expresses 
the  relationship  between  the  resources  of  a  virtual  machine  and 

the  resources  of  the  '•eel  host  machine.  Cor.seauant  ly,  the  Vhrrap 
is  a  notion  which  exists  completely  independently  of  toe  orocess 
nap  or*  for  that  matter*  ary  software  visible  process  sir-ucture. 
iepc.r3tion  and  isolation  of  the  resource  mao  (v^mao)  has  often 
seen  confused  or  overlooked  by  previous  authors  t>l*7c*108]»  3rd 
this  perhaps  accounts  for  the  l3cK  of  success  irs  designing 
easv-to-orogram  vi rtua I izafc  le  architectures. 
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A  resource  name  used  bv  a  virtual  computer  system  will  ba 
called  5  virtual  name*  virtual  resource  name*  or  v-n3me»  ard  the 
set  of  virtual  names  defines  a  virtual  space  or  environment.  A 
resource  name  used  by  the  ohysical  hardware  is  called  a  real 
name*  real  resource*  or  ^-name*  ana  the  set  of  real  names  in 
called  the  real  space  or  environment. 

In  the  resource  soaces*  real  and  virtual*  we  must  include 
the  processor*  memory*  170  system  and  any  connections  between 


them.  For  example*  resource  names  must  exist  for  every 
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addressable  unit  (bit*  byte,  word,  etc.)  ir  the  physical  Temcry. 
Resource  nares  must  exist  for  every  occoce  or  the  processor. 
Names  are  also  needed  fcr  all  I/O  cevices. 

Therefore,  the  virtual  computer  system  construction  requires 
a  mapping  function  that  will  mao  the  complete  -set  of  virtual 
resource  names  into  their  correspcrdi ng  physics!  resource  rames. 
This  mapping  function  is  illustrated  in  Figure  Mo  a  criori 
sssumotion  is  made  about  the  nature  of  the  maooirg  function  f. 
[However,  the  implementation  oart  of  the  VM  definition  [Section 
2.11  implies  that  once  the  manning  is  established  most 
instructions  of  the  virtual  machine  execute  directly  or  tne 
nost.l  The  mapping  functior  f  is  callec  the  virtjal  machine  irap* 
resource  mar,  or  VMmap. 

For  future  reference*  we  denote  the  virtual  space  names  by 
V'-TvOtvl*.  .  .val  anc  the  real  soace  names  by  R  =  {»*n  ,r  , ,  .rfc>.  V 
is  the  resource  soace  of  the  virtual  machine  3rd  9  is  the 


resource  soace  of  the  host.  We  maKe  no  assumption*  here*  about 


the  relative  sizes  cf  V  and  9.  9oth  V  and  P  include  the  memory 
space  names.  But  they  also  include  the  other  resources*  as  well. 
For  example,  we  can  imaqine  ordering  the  opcodes  by  assigning 
them  the  names  following  the  memory  soace  names  in  V  and  P. 
Lauer  and  Snow  [?S]  have  observed  that  it  may  do  sufficient  to 
consider  only  the  memory  space  names  since  other  resources  are 
often  codec  to  appear  as  memory.  For  example*  tney  observe  that 
the  accumulators  of  the  .TEC  PDP-1C  and  the  I/O  devices  of  the  CEC 
°0F-11  are  each  addressable  as  memory  locations. 

Since  we  assume  no  3  priori  correspondence  between  virtual 
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f  :  V  — »•  R  U  {t} 

f  (y)  =  fz  if  z  is  the  real  name  for  virtucl  name  y 

t  if  y  does  not  have  a  corresponding  real  name 


( level  n  +  1 ) 


( level  n ) 


Examples  ; 

f  (6142)  =  12142 

f  (25000)  =  t  if  25000  out  of  virtual  memory 

f  (processor  3)  =  processor  2 

if  virtual  processor  3  is  real  processor  2 

f  (processor  2)  =  t 

if  virtual  processor  2  not  assigned 
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me  res!  names*  we  must  incorporate  a  way  of  associating  virtual 
names  with  real  names  during  execution  of  the  virtual  rachine. 
To  this  end,  we  define,  for  each  moment  of  time,  a  function 
ft  V - >R  U  Ct> 

such  that  i f  y  e  V  and  z  e  P  then 

f  (y )  -  Cz  if  ?  is  the  real  name  for  virtual  name  y 
t 

Ct  if  y  does  not  have  a  corr  escort  ding  real  name 


fhe  value  f  f  y  >  = 1  causes  t  trap  or  fault  to  some  fault  handling 
orocedure  in  the  machine  R.  For  clarity  we  always  term  a  VM 
fault  fault,  never  an  exception.  The  spaces  V  eno  R  define  two 
virtual  machine  levels.  We  say  that  the  virtual  machine  V  is  at 
3  higher  level  than  the  machine  R.  If  the  level  of  R  is  n,  then 
the  level  of  V  is  n  +  1.  If  p  is  the  physical  machine  then  it  is 
level  0.  Thus,  a  VK-fault  is  a  VM-levet  fault  a^a  cortrol  passes 
to  a  fault  handling  machine  at  the  next  lower  level.  See  Figure 
+  -<♦. 


iycc.lna  a  vintyjJL  MsciUns*  i  °  4 

In  IV  generation  systems,  running  the  virtual  machine 
V  =  TvC  ,vl, . .  ,vb>  implies  running  some  process  P=CdC,o1. • .pa}  on 
the  virtual  machine.  This  defines  the  trapping 
ft:  P - >v  U  te> 

as  before,  with  virtual  resource  names,  V,  substituted  for  real 
ones  in  the  resource  range  of  the  mao.  See  Figure  4-5a. 

The  virtual  resource  names,  in  turn,  are  maocec  into  their 
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real  equivalents  by  the  nap*  f*V--->R..  See  Figure  4-5b.  Thus,  a 
orocess  nanre  x  corresponds  to  a  real  resource  f(4(x)).  See 
Figure  4-f>c.  In  general*  process  names  are  Tanned  into  real 
resource  names  under  the  (composed)  nap 
f  o  4>  P - >  R  U  Ct}  U  Ce.  . 

Section  4.6  provides  examples  which  illustrate  msooing 
process  names  under  the  composed  mao  f  o  4. 

There  is  no  a  priori  reauirement  that  f  or  4  be  of  a 
particular  form  or  that  there  be  a  fixed  relationship  between 
their.  (However*  in  some  sense*  4  corresponds  to  how  the  users 
would  like  to  program  the  machine  while  f  corresponds  to  how  the 
system  would  like  to  utilize  its  resources.]  we  discuss  this 
point  further  in  Section  4.6.  We  have  insicated  that  in  IV 
generation  systems*  the  memory  component  of  the  map  4  will  likely 
be  segmented.  Since  f  maps  resource  spaces*  its  memory  component 
cannot  be  segmented.  The  only  requirement  of  f  3nd  4  is  that  the 
range  of  4  be  included  in  the  domain  of  f.  (Figure  4-5c) 

We  further  distinguish  between  f  and  4  by  contrasting  the 
notions  of  level  and  layer  in  IV  generation  virtual  computer 
systems.  As  previously  stated  f  establishes  a  level  relationship 
between  a  virtual  machine  and  its  corresponding  real  machine. 
Thus*  f  exists  as  an  interlevel  map.  On  the  other  hand*  *  is  an 
intra-level  map,  mapping  process  names  into  a  oarticular  level  of 
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resource  names.  One  characteristic  of  the  £  mao  in  I V  generation 
systems  will  be  layering.  Thus  a  IV  generation  VCS  will  be 
multi-layer  per  level.  [See  also  Section  2.1.1 

In  the  event  of  a  IV  generation  VCS  process  name  exception 
control  should  be  given  to  a  more  privileged  layer  within  the 
same  level.  A  virtual  name  fault*  however#  should  cause  a  fault 
to  a  orocess  in  a  lower  level  number  machine.  Sea  Figure  4-6. 

Another  contrast  between  f  and  A  concerns  their  visibility 
oy  software  procedures.  Since  A  is  an  intralevel  mao*  the  inner 
layer  privileged  software  of  that  level  will  liKely  manipulate  A. 
However*  f  is  an  interlevel  map.  There  should  be  rjo  way  in  which 
software  at  level  n+1  (regardless  of  layer)  can  access  the  f 
which  maos  level  n+1  into  level  n.  Thus*  the  f-map  is  jnv isib  le 
to  all  software  at  level  n+1 


Iyqs  II  lirluai  Marines  2syi_sii£<3 

As  discussed  in  "Virtual  Machines*  Semantics  and  Examples” 
[53],  and  Chapters  2  and  3,  a  Type  II  (extended  machine  host) 
virtual  machine  organization  is  one  in  which  the  VMM  runs,  not  on 
a  bare  machine  as  the  supervisor,  but  rather  on  an  extended  host 
under  the  host  supervisor.  Depending  upon  tne  uses  of  virtual 
machines*  whether  or  not  there  exists  one  preferred*  favored 
operating  system,  and  whether  that  operating  system  has  certain 
capabilities  To  ocrmit  its  construction*  a  Type  II  virtual 
machine  system  can  be  a  useful  and  possioly  easy  to  implement 
artifact. 
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There  is  a  very  similar*  corresponding  conceot  to  Type  II 

systems  in  IV  generation  virtual  machines.  In  a  Type  II  IV 

generation  system,  there  is  a  slightly  different  V* mao  f*  which 

naps  virtual  resource  names  into  corresponding  process  names  for 

soire  process  running  on  the  real  machine.  Thus.  for  virtual 

names  V,  as  oefore.  and  for  process  names  Pr.  fcr  a  process 

running  on  the  real  machine,  we  define,  for  each  moment  of  time, 

a  new  function 

f  • :  v - >  Pr  U  Ct} 

such  that  if  y  e  V  and  z  e  Pr  then 

f*(y)  =  tz  if  z  is  the  real  P-name  for  V-name  y 
C 

Ct  if  y  does  not  have  a  corresponding  P-name 


The  value  f*(y)=t  causes  a  VM  fault  to  occur  to  some  fault 
narclirg  procedure  in  the  process  Pr  Cor  sore  other  orocuss 
cooperating  with  Pr)« 

Since  the  process  Pr  is  defined  on  the  real  machine  via  (its 
system  base)  the  function  4r,  a  P-name  generated  by  the  VMirap  is 
napped  into  the  corresponding  R-name.  Thus,  we  nave,  for  V-nome 
y»  the  R-rame  $r(fMy)).  This  mapping  is  illustrated  in  Figure 


4-7. 


Now  for  any  process  Pv  running  on  the  virtual  machine  there 


is  defined  (via  the  virtual  system  base)  the  junction  $v,  which 
maps  the  set  of  P-names.  Pv,  into  their  corresponding  V-names.  V. 
Thus,  the  action  of  running  the  process  Pv  causas  a  P-name,  x,  to 
be  mapped  into  the  R-name  4r ( f *  (4v  (x)  ) )  •  This  mapping  is 
illustrated  in  Figure  4-5. 
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TYPE  E  VMmt  p 
FIGURE  4-7 
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Observe  that  in  a  Type  II  system*  there  exists  a  VM  map  f* 


for  each  process  which  creates  a  virtual  machine  (or  machines)  at 
e  higher  level  number.  3y  invoking  the  layering  of  the  real 
machine  we  can  control  the  ability  of  processes  to  create  virtual 
machines  and  if  desired  can  legislate  that  there  be  only  one. 

Section  4.6  IE  ample  8]  provides  a  detailed  example  o'  a 
Type  II  IV  generation  VCS.  However*  a  simple  example*  here*  may 
show  the  intent  of  this  construction.  Let  f*  (x)  =  3lx.  Then  the 


#j3 


Type  II  VNmap's  memory  component  maps  the  virtual  machine's 
virtual  memory  into  one  segment*  say  3*  of  the  process.  This 
permits  the  normal  operating  system  to  run  3S  the  host  cn  the 
real  machine  and  perform  resource  allocation  for  the  virtual 
machine  (on  a  segment  basis). 


Recursion 

As  discussed  in  "Virtua I izeab I e  Architectures"  [51]  and 
Chaoter  2,  recursion  for  virtual  systems  is  more  than  a  matter  of 
conceptual  elegance  or  a  consideration  of  logical  closure. 
Indeed*  it  is  a  capability  of  considerable  Practical  interest. 
In  its  simolest  form,  the  notion  of  recursion  for  virtual 
machines  is  that,  although  it  makes  sense  to  run  standard 
processes  or  the  virtual  machine,  in  order  to  test  out  the  VMM 
software  on  a  VM  it  is  also  necessary  to  be  able  to  run  at  least 
3  level  2  virtual  machine.  The  real  machine  is  level  0. 

In  the  discussion  which  follows*  we  use  a  Cewey-decima I  tree 
naming  convention  in  which  a  virtual  machine  at  level  n  has  n 


corresponding  VMmao*  e.g. 
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In  a  Type  I  IV  Generation  VCS*  if 

fit  Vi - >  9 

fl.lt  Vl.i - >  Vi 

41  P - >  vi.l 

then  the  action  of  running  the  process  P  causes  the  P-nane  x  to 
be  trapped  into  f  i  (  f  1. 1  (*<x> ) )  or  fl  o  fl.l  o  6  (xt.  See  Figure 

4-9. 

In  this  function  fl  o  U.l  o  *«  we  identify  three  possible 
faults  or  exceptions? 

(1)  The  process  exceotion  to  an  inner  layer  procedure 
of  level  ?»  i.e.  4(x)=e» 

(2)  The  level  2  resource  (virtual  machine)  f3Ult  tc  an 
inner  layer  procedure  of  level  if  i.e.  fi.l  o  *(x)=t. 

(3)  The  level  1  resource  (virtual  machine)  fault  to  an 
inner  layer  procedure  of  level  0  (the  real  machine)* 
i.e.  fl  o  fl.l  o  t(x)=t. 

For  the  general  case  of  level  n  recursion,  we  have  P-name  x 
oeing  mapped  into 
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fl  o  fl.l  o  ...  o  fl.  ...  .1  o  4  fx) 

Thus*  for  a  Type  T  V M,  regardless  of  the  level  of  recursion* 
there  is  only  one  aoolication  of  the  mao  4  followed  by  n 
applications  of  an  f  map*  This  is  an  imoortant  result  that  cotes 
ou*  o f  tne  formalism  cl  distinguishing  the  f  end  6  naos.  An 
obvious  result  to  be  discussed  later  is  that  in  a  system  with  a 
complicated  4  but  with  a  simple  f*  n-level  recursion  may  be  easy 
end  inexpensive  to  imp  I  enen.t . 

•ly  analogy*  we  define  Type  II  VM  recursion*  See  Figure 
♦-10.  If 

4  t  Pr - >R 

fix  VI - >  Pr 

41!  PI - >  VI 

fl.l!  VI. 1  — >  PI 
*1.1!  Pl.l  - -  VI. 1 

Then  the  action  of  running  the  process  Pl.l  causes  the  P-name  x 
to  be  mapped  into  41.  o  f  1  c  41  o  fl.l  o  41.1  (x). 


TYPE  I  RECURSION 
FIGURE  4-9 


TYPE  JL  RECURSION 
FIGURE  4-10 
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4.3  UNSUITABILITY  CF  SCFTkARF.  f 


Before  dismissing  the  oossioility,  it  will  be  necessary  to 
scrutinize  IV  generation  systems  to  see  if  a  software 
implementation  of  the  VKmap  f»  is  viable.  ke  will  see  if  the 
techniaues  of  the  III  generation  VCS  can  be  aool ied  to  the  IV 
g enerat ion. 

As  has  been  developed  in  Chapter  3*  a  III  generation  VCS  is 
normally  constructeo  by  s 

(1)  Using  the  available  memory  mapoing  hardware  to 
help  manage  the  memory  of  the  virtual  machine*  i.e. 
virtual  memory. 

(2)  Mapping  the  master  and  ^lave  modes  of  the  virtual 
machine  into  the  slave  mode  of  the  physical  host 
machine  ana  simulating  the  effect  of  privileged 
instructions  by  the  VMM. 

(3)  Trapping  all  I/O  instructions  ana  mapoing  virtual 
device  names  into  real  devices  via  software  in  the  VMM* 
T Wv  will  ignore  1/0.3 

Three  of  the  architectural  characteristics  that  one  might 
expect  to  be  useful  in  a  software  implementation  of  f  on  a  IV 
generation  system  aret 

(1)  Segmented  address  space 

(2)  Rings 

{3)  Emulation. 

One  might  expect  the  address  mapping  to  be  useful  in 
establishing  the  virtual  memory  while  the  ring  orotection  system 
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niqht  orovide  the  layered  access  control  neeceo  to  isolate  the 
VMP  from  its  subject  VM.  As  a  last  resort*  one  might  expect  to 
fall  back  on  the  extensive  emulation  facilities  of  IV  generation 
systems  t5*i;.9J. 

Unf ortunatel y *  as  suggested  in  Section  4.2*  none  of  these 
features  helps  the  imo I ementatior  of  a  VCS .  Indeed*  they  make 
virtualization  potentially  more  difficult  since  they  will  have  to 
oe  virtualized  as  well.  [This  difficulty  is  similar  to  the 
problem  noted  by  Cheatham  1253  regarding  the  se I f -da f ini t ion  of 
particularly  rich  programming  languages.]  The  anoblem  results 
because  segmentation*  rings*  and  emulation  are  visible*  i.e.  they 
are  part  of  the  interior  decor*  and  are  represented  by  constructs 
in  the  system  base  which  may  be  seen  an:  manipulated  by 
("privileged"  ring  i)  software  procedures.  We  can  illustrate 
this  difficulty  with  a  look  at  rings* 


Ssciilin*  q1  Hinas 

A  software  implementation  of  the  VMmap  f  on  a  IV  generation 
VCS  reauires  (1)  maintenance  of  a  virtual  system  base  by  the 
virtual  machine  monitor*  and  (2)  the  control  cf  access  to  the 
real  system  base  by  instructions  of  the  virtual  machine  1511.  By 
analogy  with  the  III  generation  "software  VMmac"*  we  orevent 
access  to  the  real  system  base  by  not  running  ring  0  cf  the 
virtual  machine  as  ring  •]  of  the  rest  machine.  Into  what  ring 
can  ring  0  be  mapped? 

Two  of  the  properties  of  rings  which  must  be  creserved  in 
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jny  mapping  are 


(1)  monotonicity 

(2)  finiteness. 

That  is*  the  rings  define  an  ordering  of  privilege:  ring  1  is 

more  privileged  than  ring  |  if  i<]  [57, 1061.  Furthermore*  there 

is  a  finite*  relatively  small  number  of  rings  in  the  system’s 

universe*  e.g.  Multics  has  eight  135*1061. 

Therefore*  two  ring  mapoinq  schemes  are  oresentej  in  Figure 

4-11.  The  first  mapping  scheme, 

f  (x)  =  Cx  if  3<x=<m 
t 

Ct  if  x=C 

maps  rings  identically  except  in  the  most  privileged  ring.  All 

ring  0  instructions  are  simulated  on  an  instruction  by 

instruction  basis.  An  attempt  by  an  outer  ring  to  call  a  ring  0 

orocedure  must  cause  a  process  exception  «hich  is  understood  as  a 

fault  to  the  VMM.  This  construction  is  the  IV  generation 

equivalent  of  the  hybrid  virtual  machine*  HVM .  ISee  Sections  2.1 

and  3.3.1  The  empirical  requirements  are  similar,  e.g.  attempted 

calls  to  ring  3  can  be  made  to  fault.  The  performance  of  the  IV 

generation  HVM  is  significantly  degraded  because  of  the  likely 

frecuent  software  interpretation  of  all  ring  instructions. 

The  second  mapping  scheme, 

f  (x)  =  Cx*l  if  G  =  <x=<i  for  some  i=<m 

C 

Cx  if  i<x=<m 

maps  ring  C  into  ring  1.  To  preserve  monotonicity  of  rings*  ring 
t  must  be  mapped  into  ring  2*  etc.  However*  because  of  the 
finiteness  property*  it  is  necessary  to  map  a  (particular)  oair 
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Ring  mapping  scheme  2* 


if  0<x<i 
if  i  <x<m 


RING  MAPPiNG  SCHEMES 
FIGURE  4-11 


This  is  rather  difficult  because  of  the  potential  visibility  of 
the  ring  number  in  the  instruction  counter  ('ring  register 
field*!  or  in  stacks*  for  calls  across  a  ring  boundary  [1061. 

another  problem  which  might  develop  concerns  the  absolute 
interpretation  of  physical  ring  number  for  certain  side  effects. 
For  example,  one  proposal  11061  has  *he  hardware  identify  ring 
numbers  with  the  segment  number  of  the  stack  segment  for  that 
ring,  or  ring  numbers,  i.e.  0*1,  2-5,  6-7,  with  different  access 
properties  to  gates. 

One  simple  hardware  proposal  that  might  pe  m3de  to  avoid 
some  Pitfalls  of  the  second  mapping  scheme  is  a  ring  relocation 
register.  See  Figure  4-12.  The  ring  relocation  register  could 
oe  used  to  linearly  displace  ring  numbers  and,  thus,  crevent  ring 
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implementation  of  IV  generation  virtual  machines.  As  we  shall 
see*  the  software  mapping  techniaue  is  unrat  oral  and  unnecessary 
given  a  proper  underst anting  of  the  mode!  of  Section  <*.2,  Fourth 
generation  systems  formalize  the  process  map  in  firmware.  Thus* 
in  introducing  virtual  machines  into  the  Itf  generation*  it  is 
natural  to  consider  how  firmware  may  be  used  to  formalize  the 
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4.4  THE  VENICE  PAPER 


8L 

r* 


fc 

<S, 

! 

& 

I 


£ 

iK 

S? 

I 

1 

& 


Et 


K 


In  "Virtual ize3b I e  Architectures"  [51!»  called  the  Venice 
°aper  or  Venice  Proposal,  abbreviated  as  VP.  the  authors  propose 
an  interirr  solution  to  the  problem  of  virtualization  of  IV 
generation  computer  systems.  The  VF  observes  th3t  because  of  the 
nigh  frequency  of  references  to  the  system  base,  its  maintenance 
as  pure  software  construct  is  likely  to  be  doomed  to  failure. 
[See  Section  4.3  above.!  Thus,  the  solution  orooosed  by  the  VP  is 
that  hardware-f irmware  be  provided  to  assist  virtualization. 
Each  virtual  machine  will  maintain  its  own  system  base.  visible 
to  itself.  However,  when  the  virtual  machine  is  actually 
dispatched  it  will  be  running  with  a  special  "translated  system 
base"  maintained  for  each  virtual  machine  by  the  VMM.  Thus,  any 
attempt  by  software  running  on  the  virtual  machine  to  read  its 
system  base  will  be  directed  to  the  virtual  system  base  and  will 
go  unintercepted.  However,  any  attempt  by  the  virtual  machine  to 
write  its  system  base  faults  to  the  VMM  which  can  make  the 
corresponding  change  to  the  translated  system  base  before 
redispatching  the  virtual  machine. 

Thus,  the  Venice  Proposal  provides  a  major  improvement  over 
orevious  virtual  system  designs  in  permitting  reaa  access  to  the 
system  base  to  proceed  without  interruption.  Only  on  attempted 
write  access  to  the  system  base  will  the  familiar 
f ault-simulate-redisoatch  sequence  be  reauired. 

Furthermore.  the  VP  provides  a  virtualization  design  that 
permits  a  limited  two  level  recursion  of  virtual  machines  to 


*—•— ^*~~**~  1'^^'  a  r~tT.  la 


AG&SSSSragsew: 


Page  115 


operate  with  hardware  dispatching  of  the  aocropriate  parent 
virtual  Machine  on  a  write-access  fault  to  an  irferior  virtual 
nachine  system  base.  The  performance  of  the  second  level  virtual 
machine  could  be  comparable  to  the  one  level  case. 

We  can  interpret  the  VP  in  terms  of  the  model  developed  in 
Section  4.2.  Let  P,V*R,  4*  and  f  be  as  before.  Then  the  VP 
maintains*  for  a  virtual  machine*  a  translated  system  base* 

4t  =  f  o  4 

and  this  is*  in  fact*  the  process  map  that  is  active  when  a 
process  of  the  VM  is  dispatched.  See  Figure  4-13.  The  VMmap  f 
is  strictly  a  software  trap  and  never  appears  in  firmware  accessed 
tables*  4t  is  accessed  by  the  firmware.  The  VHP  statically 
composes  f  with  4  to  produce  4t.  Thus*  f  o  4  is  "cal  cu  I  ated"  in 
advance  of  the  process  being  dispatched.  Consequently  a 
modification  to  4  cannot  be  reflected  in  a  corresoonding  dynamic 
modification  to  4t.  To  maintain  integrity*  a  modification  to  4 
must  cause  an  immediate  VM-fault  to  the  VMM*  which  calculates  a 
new  value  for  4t  and  redispatches  the  VM. 

Note  that  this  analysis  (as  with  the  discussion  of  Section 
4.3)  implies  that  there  must  be  an  easy  way  for  the  hardware  to 
tell  that  an  attempt  is  being  made  to  modify  the  system  base.  A 
special  instruction  might  be  reserved.  [See  Section  3.4.1 
Otherwise,  all  addresses  in  a  physically  "cverlayed"  segment 
structure  might  have  to  be  checked*  significantly  degrading 
oer formance . 

A  detailed  illustration  of  actual  instruction  executions  in 


the  VP  is  presented  in  Section  4.6 
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In  the  recursive  case#  tne  VP's  VMM  statically  calculates* 
via  software,  the  translated  composed  map 
tti.i  =  fl  o  it  1  =  fl  o  fl.l  o  4> 

and  this  map  is*  in  fact,  active  when  a  procass  cf  the  VM  is 
dispatched.  3ee  Figure  4-14.  Again,  attempted  write  access  by  a 
orocess  to  4  must  VM  fault  to  allow  the  change  to  be  reflected, 
via  software,  in  Atl.l. 

Susmac* 

The  VP  proposes  an  implementation  of  a  IV  Generation  VCS. 
Unlike  the  purely  software  techniaues  discussed  in  Seclion  4.3, 
the  VP  does  not  require  the  "compress! on"  of  privileged  layers  of 
the  real  machine  to  less  privileged  layers  of  the  host. 

Consequently  the  VP  should  produce  efficient  IV  generation 
virtual  machines.  Furthermore ,  the  VP  should  be  feasible  ir  many 
cases  in  which  pure  software  techniques  are  imoossible  tc  use. 
However,  the  VP  suffers  from  the  following  di saov-ntages : 

(1)  Performance  degradation  due  to  faults  on  system 

base  write  instructions 

(2)  Storage  wasted  due  to  translated  system  base.  This 
could  be  particularly  significant  in  the  recursion 
example. 

(3)  Complexity  and  software  needed,  in  the  VMM,  for  (1) 
and  ( 2) • 

In  The  resf  of  this  chapter,  we  shall  consider  a  proposed 
design  which  eliminates  these  weaknesses. 
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The  Hardware  Virlualizer  (HV)  is  a  h  srdwar  a- f  i  rmware 
implementation  of  the  Vhmap  f  < for  a  XV  generation  VCS)  »  In  this 
section,  we  shall  dtvelco  the  arguments  in  favo''  of  ar  HV,  snow 
hew  an  HV  can  be  constructed,  show  that  the  construction  is 
feasible  in  terms  of  IV  generation  hardware  (realization) 
components,  and  indicate  that  satisfactory  performance  will 
occur.  As  in  Section  A. 1-4. A  above,  because  tne  IV  generation 
-reatment  of  I/O  is  so  similar  to  III,  we  do  not  ir.ccroorate  T/0 
into  the  HV.  Similarly,  because  of  the  intrinsic  problems  with 
time  in  a  VCS  we  do  not  discuss  time.  These  difficulties  are 
considered  in  Section  A. 7. 

AnasiEjec!  lac  hv 

It  is  our  claim  that  software  construction  of  the  VMmap  f  is 


k 
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It 
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unnatural  and  unnecessary  given  3  oroper  understanding  of 
virtualization  as  developed  in  the  thesis  resoeci3lly  Section 
A, 21.  In  numerous  other  areas  of  computer  systems  research  as 
oasic  and  underlying  principles  have  been  develooed  or 
discovered,  ad  hoc  and  often  cumbersome  software  corstructs  nave 
oenn  replaced  by  elegant  hardware  (-firmware)  mechanisms.  Cne 
can  turn  for  confirmation  to  management  of  multi-level  memory 
i28»38»391  or  protection  rings  r57,lrS].  The  firmware 
implementation  of  the  process  model  in  IV  generation  systems 
(51,79*87]  is  a  tour-de-force,  including  the  two  previous 
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axamples  within  it  ss  special  subexamoles. 

We  claim  that  the  hardware  virtual izer  HV  rrings  about  a 
simplification  of  the  virtualization  mechanist.  The  HV 
simplifies  and  eliminates  most  of  the  VMM  software*  reduces  the 
lain  storage  requirements  of  the  VHM»  and  proouces  satisfactory 
operating  performance.  Furthermore*  it  permits  virtualization  to 
take  place  where,  using  previous  software  techniques*  it 
ordinarily  would  not  be  oossible. 


a 


I 


m. 
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Our  design  of  a  hardware  virtualizer  follows  directly  from 
the  model  of  Section  ««.2.  The  maps  f  and  *  are  distinct  in  a  IV 
generation  VCS.  Therefore,  we  implement  them  as  completely 
different  constructs,  caoable  of  being  referenced  and  manipulated 
independently.  Execution  by  the  virtual  machine  causes  the  macs* 
f  and  *♦  to  be  composed  dynamical  ly  to  form  f  o  I  at  execution 
time. 


The  construction  of  an  HV  must  address  the  following  points; 

(1)  The  database  to  store  f 

(2)  A  mechanism  To  invoke  f 

(3)  The  action  on  a  VM-fault 

In  the  discussion  which  follows*  we  shall  develop  fre  basis  for 
an  HV  design  somewhat  independently  of  the  particular  form  of  the 
VMmap  f  which  is  being  implemented.  Although  we  shall  refer  to 
certair  particular  f  structures*  such  as  the  3-3  or  paging  form 
of  memory  map*  the  actual  detailed  examples  are  postponed  until 
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Section  4.6..  [The  discussion  which  follows  assumes  the  IV 


generation  orocess  map  4  presented  in  Section  4.1.  As  we  note  in 


Section  4.3*  suitable  simo  I  i  f  i  cat  i  on  anc  int  epcr*  t  at  i  on  of  ♦he 


*-nap  allows  us  to  extend  this  construction  to  second  and  tni^d 


generation  architec tures  as  well.] 
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The  Vkk  must  create  and  maintain  a  database  which  represents 


the  VMirap  f  for  the  next  level  virtual  rr.schin els).  This  database 


must  be  stored  such  that  it  is  invisible  to  the  virtual  machtre* 


including  its  most  privileged  software.  Let  us  assume  that  for 


economic  reasons  [511  the  database  must  be  stored  in  main  memory. 


Then  the  VMtrap  itself  must  be  such  that  f  is  not  in  the  (virtual) 


memory  of  the  virtual  machine. 


The  VM  database*  logically  should  be  an  addition  tc  the 


system  base*  A  pointer  at  a  fixeo  location  from  the  RCOT  points 


to  a  Virtual  Computer  System  Table  (VCSTAR).  Esch  entry  in  the 


VCSTAB  points  to  a  Virtual  Computer  System  Control  BlocK  (VCSCB) 


or  colloquially,  VMC3.  Thus*  the  i-th  entry  points  to  the 


control  block  for  virtual  computer  system  i. 


The  VCSCB  provides  the  implementation  of  tne  Vkmsp  f  for  the 


virtual  machine.  It  contains  the  memory  map*  orocesssr(s)  map* 


and,  at  least  conceptually*  the  I/O  map.  In  addition*  there  may 


be  other  status  and/or  accounting  data  for  the  virtual  machine. 


As  noted  in  the  VCS  model  (Section  4.2),  the  maos  4  and  f 


are  completely  different.  In  particular?  the  texory  components 
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of  the  maps  serve  different  purposes  and  so  are  likely  to  be  of 


different  form.  The  i  trap  linelv  wilt  be  segmented.  Depending 


upon  how  the  VM*s  will  be  used*  the  f  iiijd  might  oe  R-3*  oaoed,  or 


nu  !  ti - ! eve  I  paged.  If  s  limited  number  of  virtual  systems  are 


used  irregularly  for  operating  system  debugging  or  subsystem 


transferabi li ty,  ar  R-3  map  might  be  reasonable.  If  fictitious 


memory  resources  are  raauired.  some  form  of  cage  map  may  be  used. 


I  See  pert or  mane  e/cost  discussion  below.]  When  we  oiscuss  a  page 


-nap,  we*  usually*  do  not  illustrate  fictitious  resources*  e.g, 


demand  paging. 


The  processor  componenl  of  the  VCSCB  includes  (£>fic  processor 


of  the  VC5)  the  saved  values  of  the  processor  ragisters  (the  last 


time  the  VM  was  running].  Note  that  this  is  different  from,  the 


per  ccflfiSSS  saved  register  values  kept  in  the  resoective  PCB*s  of 


each  process  on  the  Vh. 


These  registers  typically  might  include: 


P  -  current  process  ID 


IC  -  instruction  counter 


T  -  top  of  stack 


STR  -  status 


BR  *•-  base  registers 


GP  -  general  registers 


SR  -  scientific  registers  (floating  point) 


fin  this  discussion*  we  assume  that  ROOT  is  zero  for  simplicity.) 


Additional  processor  related  information  wept  in  tne  VRC3 


includes  capability  information,  for  the  particular  virtual 


processor  (not  the  real  processor  and  net  processes  running  on 
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the  virtual  rrocesscr)  irdicst, pc  rarticulsr  festu^es  and 
i ns truct ions,  oresert  or  absent.  These  include*  for  example* 
jcirntific  instruction  set,  Vk  instruction  s^t  (LV^IO 
i  f's  t  rue  t  inr  I ,  etc.  Additional  information  micht  include  a 
iererr-1  orerr?  fault  rrts-t  or  a  fictitious  resource  mask.  The 
oncefp  mask  can  be  used  to  force  certain  odcoo?s  in  the  Vk  to 
It  tc  the  IT^is  rction  is  cenived  from  Section  3.k.)  A 

fictitious  resource  mask  might  indicate  whether  (bounds  tyce) 
faults  are  to  he  directed  to  the  V*  or  the  VMk, 
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Tn  order  to  invoke  the  VkrrsD  f*  the  hv  requires  cne 
additiorsl  visible  reqister  ard  cne  instruction  for  mar ipu lat ing 
it.  The.  register  is  the  VklO  and  contains  the  rumb^r  of  the  VCS 
in  the  VCSTAF  that  is  currently  executing.  T^e  new  instruction 
is  lOAC  VkTC,  L VMTC .  Sea  flowchart  in  Fiqure  k-15. 

For  the  hardware  virtuelizer  desigr  to  he  successful,  the 
'/mic  register  (and  the  LVMic  instruction)  must  have  three  crucial 
orcrerties  151). 

(1)  The  V*TC  register  absolute  conterts  nay  neither  be 
read  ror  written  hy  software. 

(2)  The  only  modifications  possible  tc  the  value  in 
the  VWIO  register  are  accerding  a  low  order  IC  syllable 
(as  a  result  of  an  IVMID  instruction)  or  removing  some 
number  of  low  order  ID  syllables  (as  a  result  of  a  VM-fault). 

(?)  The  VMO  of  the  real  rrachire  is  the  null 
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MEMORY  MAP 
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ident if ier. 


These  crooert ies  all  follOM  directly  from  the  dasire  to  model  the 
relative  resource  contexts  provided  by  invocation  of  f. 

Meedless  to  say*  LVMIO  “loads"  (appends)  the  VXIO  with  the 
10  specified  as  the  operand  of  the  instruction.  If  the  VM 
capability  does  not  exist  for  this  machine,  a  VH  capability 
exception  occurs*  which  is  a  process  exception  to  a  orivileged 
procedure  in  the  same  machine.  If  the  \/MID  is  loaced 
successfully*  the  firmware  checks  the  appropriate  VCST4B  entry 
for  validity.  If  it  is  marked  as  absent  or  invalid*  a  VCSTflB 
absent  or  invalid  exception  occurs  to  a  procedure  in  the  same 
machine. 

Otherwise  the  processor  enters  virtual  mode  and  (assuming  no 
faulting  because  of  VCSC8  values)  loads  the  processor  registers 
(P»  IC»  BR,  etc.)  from  the  VCSCB. 

Now  the  memo~y  component  of  f  is  active*  ssy  relccatior  and 
pounds*  R-R.  This  causes  a  loading  of  the  R-B  into  scratchoad 
registers*  say*  Rcomp-Rcomp .  Thus,  every  virtual  memory  name 
will  be  transformed  to  a  real  name  by  the  R-B  map,  i.e.  value 
currently  in  Rcomp-Bcomp.  (Of  course,  the  scratchpad  registers 
^como-Bcomp*  may  be  neither  read  nor  written  by  any  software. 
When  the  real  machine  (VMID=null  identifier)  is  running*  Pcomp=0 
and  9comp=ohysica I  memory  size.)  In  a  segmentation  system,  the 
result  of  looking  up  a  segment  P-naire  and  obtaining  the  V-name 
must  be  subjected  to  the  R~B  map.  However*  since  the  segment 
table  is  not  the  root  of  the  system  base*  the  firmware  must  first 
locate  it  from  the  true  ROOT  and  the  process's  P-name*  i.e. 
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orocess-id.  This  endeavor  reauires  the  firmware  to  ma^e  several 
•-eferences  to  main  memory  in  the  course  of  "walKirg**  tha  system 
base  list  structure.  Each  of  these  references  to  tne  virtual 
lemory  must  be  mapped  to  the  real  memory  equivalent  by  applying 
3-B*  i.e.  adding  Rcomp-Rcomp. 

Eventual ly»  the  location  of  the  segment  tacl3  is  established 
and  it  is  saved  in  a  scratch  pad  register.  Thus*  any  subsequent 
memory  reference  may  directly  utilize  the  segment  table  to  obtsin 
a  virtual  memory  location  which  is  then  mapped*  via  P-R*  into  its 
real  eauivalent. 

For  reasons  of  per f ormanc e ,  the  original  machine  requires  an 
associative  memory  to  store  the  most  recent  ‘segment 
number-memory  location*  oairs.  Then*  the  hardware  vir tua I  izer *s 
asscciator  will  store  ‘segment  rumber-rea I  memory  location* 
oairs.  On  setting  a  new  VhlD*  the  associato**  can  be  cleared. 
Another  possible  choice*  which  can  be  made  on  the  basis  of 
cost/oerf ormance*  is  to  have  a  still  larger  assaciator  anc  Keep 
the  '/HID  as  a  field  to  fce  used  as  part  of  the  search  Key. 

Recursive  invocation  of  virtual  machines  adds  little 
additional  complexity  to  this  implementation.  If  the  VM 
capability  bit  in  the  VRCR  is  set  to  one*  then  the  virtual 
machine  has  the  same  (software  visible)  hardware  virtualizer  VM 
features  as  in  a  real  machine.  Therefore*  a  \Hh  rurning  or  the 
V M  may  attempt  to  create  an  inferior  virtual  machine  by  creating 
a  VCSTA8  entry  and  s  VCSCB  for  the  level  2  VM  in  its  system  base. 
It  may  also  dispatch  the  (level  2)  VM  by  executing  3n  LVMD 
instruction. 
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As  noted  in  the  Venice  Paper  [511  ana  3bove,  recursion 
reauires  that  the  VKIC  be  made  a  “multi-syllable  register".  When 
the  real  machine  is  executing,  the  VMIO  value  is  ru  1 1 .  When  the 
VMM  dispatches  a  VM,  it  loads  (appends)  a  value  into  the  VMIO, 
say  1.  If  that  virtual  machine  is  running  a  VMM  which  creates 
and  dispatches  a  VM,  say  ID  1*  then  1  gets  appended  to  the  VM.ID. 
Thus,  the  new  composite  value  in  the  VMID  is  *1.1*  which  gives 
the  ID  of  the  currently  executing  VM.  As  stated  above,  this 
value  tray  not  be  either  reaa  or  written  by  anv  software  of  the 
virtual  machine.  See  the  tree  structure  shown  in  Figure  4-16. 


To  handle  a  small,  finite  depth  of  recursion  reauires  the 
VMIO  register  to  be  sufficiently  "wide"  for  specification.  In 


practice,  one  should  not  expect  the  VM  level  to  go  more  thar  3  or 
4  deep. 

As  in  the  non-recursi ve  case,  the  execution  of  the  LVMID 
instruction  by  the  level  1  VMM  causes  its  VCSTAE  and  VCSCB  entry 
to  be  checked  for  absence  or  validity.  If  the  LVMID  completes 
successfully,  the  machine  is  now  in  virtual  mode  at  the  next 
I  eve  I , 

Now  the  memory  component  of  the  new  f  is  active.  Hr  this 
discussion,  a  map  of  the  form  R-B. I  This  causes  an  addition  of 
the  new  R-B  to  the  scratchpad  register  Rcoiro-Bcomo  where  the 
current  R-8  value  "total"  is  being  maintained.  However,  in 
addition  to  the  running  sum,  a  collection  of  scratchpad  registers 
are  set  aside  to  hold  the  R-B  values  of  each  level.  tAs  Lauer 
and  Snow  1761  observe,  it  may  also  be  possible  for  the  firmware 


to  store  the  reverse  "trail"  ir  main  memory.)  This  permits 
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restoring  the  previous  composed  map  by  subtracting  on  a  VM-fault. 
In  terms  of  the  VCS  model*  the  running  su.t.  of  the  R-B*s, 
Rcomp-Bcomp ,  provides  a  calculation  of  the  composition  of  the  two 
f  maps  (in  the  memory  dimension)*  say  fl  o  fl.l.  Then  the 
composed  mao  is  combined  with  the  4-map. 

Thus*  every  virtual  memory  name  will  be  transformed  to  a 
real  name  by  the  composite  R-8  map*  Rcomp-Bcomp.  This  is  done* 
just  as  in  the  single  level  case*  with  the  composite  P-E  map 
taking  the  place  of  the  simple  R-B  map.  This  includes  walking 
the  level  2  system  base,  etc. 

Just  as  in  the  single  level  case  where  an  associative  memory 
was  used  to  store  ‘segment  numbei — real  memory  location*  pairs*  so 
in  the  recursive  case,  an  associator  stores  ‘level  2  segment 
number-real  memory  location*  oairs.  The  performance  implications 
of  the  associator  (map  composer)  design  will  be  discussed  belcw. 
Section  4.6  provides  detailed  examples,  including  illustrations 
of  the  associator  during  HV  operation. 

Our  discussion  of  the  VM  invocation  has  utilized  an  R-B 
memory  component  in  the  VCSCB.  The  mechanism  described  works  for 
other  memory  maps*  as  well.  [However*  we  are  assuming  that  there 
is  only  one  f  map  structure,  common  to  the  entire  system.)  For 
example,  a  paged  memory  map  might  be  stored  in  the  VCSCB.  After 
the  IVMIG  instruction,  map  composition  in  the  memory  dimension  is 
dynamically  performed  on  a  page  basis.  This  implies  that  the 
associator  for  a  paged  HV  is  of  a  slightly  different  form  from 
the  R-B  HV.  Illustrations  are  provided  in  Section  4.6. 
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On  a  Vt'-fault  (VP-level  fault)*  control  must  pass  to  the 
VMM.  If  the  VMM  is  running  on  the  real  machine,  control  passes 
to  the  real  machine.  If  the  VM  is  running  at  level  n,  then 
control  passes  to  the  VMM  at  level  n-1.  The  faulting  mechanism 
(firmware)  must  inspect  the  VMIO  register  and  discard  the  low 
order  syllable.  The  resulting  value  is  the  ID  of  the  VMM's 
machine.  See  flowchart  in  Figure  4-17. 

The  characteristics  of  the  VM-fault  can  be  derived  froir  the 
empirical  work  with  III  generation  systems  [Chaster  3).  In 
particular,  the  VM-fault  must  be  completely  invisible  to  the  VM. 
The  VM  can  have  no  way  of  testing  or  discovering  whether  a  fault 
has  occurred. 

On  a  VM-fault,  the  processor's  registers  must  be  saved  in 
the  VCSC3  and  the  asscciator  must  be  ourged.  A  fault  code  can  be 
stored  in  the  VCSCB  and  the  VMM  is  started.  After  processing  the 
fault,  the  VMM  may  return  to  the  VM  by  issuing  an  LVMIQ 
instruction. 


EegSi&LJLii*  2.1  £sm$lL22lL2n 


The  HV  described  above  is  feasible  to  construct  on  a  IV 
generation  system.  The  hierarchical  levels  of  control  and  map 
composition  facilities  add  little  additional  ccmclexity  to  the 
orocess  model.  Three  characteristics  of  IV  generation 
realizations  C191  will  be  the  use  of  large  control  stores, 
elaborate  micro-logic,  and  associative  memories.  The  hardware 
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virtualizer  wilt  utilize  these  same  buildirg  blocks* 


£fiCl£CIEaQ£e  A^iUJimii.SDS  3q£  Analysis 

The  expectation  of  acceptable  performance  by  processes 
runring  or  the  virtual  system  depends*  to  seme  extent,  cn  the 
expectation  of  acceptable  performance  in  the  re3l  system.  As 
tiscussed  above,  the  process  map  *  in  IV  generation  systems  will 
likely  involve  rather  elaborate  constructs  which  must  be  accessed 
oy  the  firmware  for  every  instruction  execution.  For  example,  in 
the  case  of  segmentation,  it  will  be  necessary  for  the  firmware 
to  obtain  the  resource  location  names  (from  F-names)  via  *  for 
every  instruction.  This  will  typically  reouirei 

(1)  The  most  recently  referenced  segments  to  have 
P-name,  R-name  pairs  stored  in  some  small  finite  number 
of  associative  registers,  and 

(2)  The  recently  referenced  segments  to  be  maintained 
in  main  memory  by  the  software. 

Needless  to  say,  3ny  architecture  which  incorporates  paging 
(as  well  as  segmentation)  in  the  *  map  will  certainly  be 
implemented  this  way.  [As  we  shall  see  below,  caging  properly 
oelongs  in  the  f-map  instead.) 

That  a  IV  generation  architecture  (with  segmentation,  etc.) 
should  perform  acceptably  depends  significantly  on  the  Drincicle 
of  locality.  From  the  initial  notion  of  "program  locality"  as 
noted  by  Dennis  and  Denning,  Madnick  1601  has  generalized  and 
identified  two  specific  aspects  of  locality  that  3re  used. 
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(1)  Temooral  locality 

If  the  logical  addresses  <al*  a2*  ..♦>  are  referenced 
during  the  time  interval  t-T  to  t*  there  is  a  high 
probability  that  these  same  logical  addresses  will  be 
referenced  during  the  time  interval  t  to  t+T. 

This  behavior  can  be  rationalized  by  program  constructs 
such  as  loops*  frequently  used  variables*  and 
frequently  used  subroutines  180]. 

(2)  Spatial  locality 

If  the  logical  address  a  is  referenced  at  time  t*  there 
is  a  high  probability  that  a  logical  address  in  the 
range  a-A  to  a+A  will  be  referenced  at  time  t*l. 

This  behavior  can  be  rationalized  by  program  constructs 
such  as*  sequential  instruction  sequencing*  and  linear 
data  structures  (e.g.  arrays)  18GJ. 

In  IV  generation  systems*  because  of  the  complex  4  3nd  the  cest 
required  to  "start  up"  the  map*  it  is  JifceJy  that  the  (software) 
scheduler  and  (firmware)  dispatcher  will  enforce  an  additional 
I ocal ity* 

(3)  Process  locality 

If  the  process  identifier  of  the  process  executing  at 
time  t  is  P*  then  there  is  a  high  probability  that  it 
will  be  P  at  tine  fti. 

Virtual  machines  and  the  hardware  virtual izer  asd  a  new  notion. 
(*♦)  Virtual  machine  locality 

I*  the  VMIO  of  the  currently  executing  VM  at  time  t  is 
xl«x2.  ...  ixn-l.xn,  then  there  is  =  riot,  probability 


U 
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that  the  VMID  wit!  be  xl,x2.  ...  «xn-l.xn  at  time  t  +  i. 
Furthermore,  the  other  values  that  it  can  tahe  are 
xl.x2»  ...  .xn-i  or  xl.x2.  •*.  .  xn-1  .xr . xn  +  i  .  The 
first  value  occurs  on  a  VM-fsult,  the  second  on 
recursion.  On  multi-level  faults,  other  values  are  possible. 
v/M  locality  implies  that,  with  proper  imp  ferrer.taticr,  one 
level  or  multi-level  recursive  virtual  machines  need  not  nave 
significantly  different  performance  from  real  tachines.  Another 
way  of  phrasing  this  observation  ist 


I  sacs  cal  ana  saaiial  iaaaiii*  ml*  aaiss  iDvaciiDi* 


Regardless  of  what  a  page-size  block  is  called,  or  how  many  times 
it  gets  renamed  (via  a  VMmap)  there  is  still  an  intrinsic 
orobability  of  reference  to  it  by  the  executing  orocess.  Thus,  a 
map  composer  aided  by  an  associative  store  should  provide 
comparable  performance  in  the  virtual  machine  to  that  obtained  in 
the  real  machine. 

He  examine  the  performance  cf  3  virtual  machine  vs.  a  real 
machine  for  the  R-8  virtual izer  and  for  the  caged  virtualizer. 
Performance  of  the  HV  can  be  judged  by  a  direct  comparison  of 
instruction  execution  rates  of  the  V H  vs,  the  real  machine.  The 
use  of  a  cache,  look-ahead,  etc.  makes  the  comparison  difficult. 
However*  we  can  constrast  the  time  required  to  ceveloo  a  Physical 
memory  address  which  then  gets  sent  to  the  memories. 

In  the  R-8  virtualizer,  as  sketched  above,  let  the  following 
oaram.eters  represent  access  times. 


a  -  associative  memory 


m  -  main  memory 
s  -  scratchpad  memory 

Let  pk(R>  be  the  probability  that  a  segment  entry  Mill  be  found 
in  an  associative  memory  of  size  k,  in  the  real  machine.  Let 
pk(V)  be  the  corresponding  probability  for  our  virtual  machine. 
Then  pk(V)=pk(R).  Let  us  fix  k  so  this  probability  is  Just  p. 
Then  the  average  time  to  develop  a  physical  memory  address  is 
approximately 

flr  =  a*p  ♦  M*(l-pl  for  the  real  machine 

Av  =  a*p  ♦  (M+s)*(i-o>  for  the  virtual  machine 

Since  the  scratchpad  maintains  a  cumulative  value  of  R-B. 
Rcoirp-Bcomp,  during  recursion,  Av  is  constant  regardless  of  the 
level  VM.  Reasonable  parameter  choices  are  3=50  nsec,  H=500 
nsec*  s=50  nsec,  and  p=.99.  Then  Ar=54.5,  Av=55»  and  Ar/Av>.99. 
This  result  implies  that,  for  an  R-B  virtualizer,  regardless  of 
the  level  of  virtualization,  Av  is  approximately  equal  to  Ar. 

In  the  paged  virtualizer,  the  associator  is  of  different 
form  from  that  of  the  real  machine.  Whereas  the  real  machine’s 
associator  relates  segments  to  physical  memory,  the  associator  in 
the  paged  virtualizer  must  relate  page-size  blocks  rnithin  a 
segment  to  the  physical  location  of  the  oage-size  block. 
Therefore,  in  order  to  contrast  the  performance  of  real  and 
virtual  machines,  me  let  pk  be  the  probability  that  a  segment 
oage-size  block  entry  Milt  be  found  in  an  associative  memory  of 
size  k.  For  a  particular  value  of  pk,  k  Mill  be  larger  in  the 
segment-page  associator  than  in  the  simple  segment  associator. 
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The  parameters  a  ant  M  are  as  above.  We  assume  that  for 
each  level  V M  a  scratchpad  register  holds  the  ease  of  the  cage 
table.  Furthermore.  we  assume  that  all  scratches?  references  are 
overlapped  Hi  th  main  memory  fetches  and  so  are  ignored  for  this 
analysis.  Then 

An  =  a*pk  4  (r4i)*M*(i-pk> 

where  n  is  the  level  of  Vk.  If  we  set  n  =  0»  we  obtain  the  result 
for  the  real  machine. 

AO  =  a*pk  +  M*il-ok) 

Figure  4-18  plots  the  ratio  AO/An  as  a  function  cf  r.  for 
3=50  nsec.  M=5Q0  nsec  and  representative  values  of  pk.  In 
oractice  n  is  unlikely  to  be  greater  than  3  or  4.  Figure  4-19 
olots  AO/An  as  a  function  o»  pk  for  n=i»2,3.4. 

From  the  graphs,  it  can  be  seen  that  performance  of  a  paged 
HV  Mill  be  respectable.  depending  upon  the  environment  and 
application,  for  a  range  of  values  cf  pk.  For  n=l. 
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Through  choice  of  H.  pk  can  likely  be  made  to  be  .99  £105}, 
Then,  the  virtual  machine  runs  at  92  percent  the  speed  of  the 
real  machine  lat  least  as  far  as  adoress  development  goes).  This 
speed  might  be  satisfactory  for  a  normal  production  environment. 

With  pk  =  .99.  a  level  2  V*  runs  with  AG/'12=.*,4.  This  is 
likely  to  be  satisfactory  for  debugging  the  VMM, 
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4.6  DETAILED  ILLUSTRATIONS 

In  the  previous  sections*  we  have  indicated  the  structure 
ana  operation  of  typical  IV  generation  hardware  v ir tua I  izers. 
The  intent  of  the  present  section  is  to  make  this  operation 
oerfectly  clear  by  illustrating  with  a  number  of  detailed 
concrete  examples  in  which  we  shew  the  execution  of  actual 
instructions  by  the  VI.  Most  of  the  examples  provide  the  same 
seauence  of  instructions  being  executed  by  (similar  or)  identical 
orocesses  on  the  virtual  machines.  This  is  done  so  as  to  provide 
a  basis  of  comparison  between  the  different  VMmBDS.  The  examples 
generally  use  stylized  instructions*  i.e.  LOAD,  STORE*  P»  V, 
S IGNAL--PROCESSOR*  and  do  not  worry  about  accumulators*  base 
registers,  etc.  Furthermore*  the  examples  co  not  cover 
fictitious  (memory)  resources,  I/O*  time  and  other  potential 
special  difficulties.  The  claim  is  that  CPU-memory  processes  are 
sufficiently  interesting  and  complex  in  IV  generation  systems. 
If  we  can  run  them  well  in  a  virtual  machine  we  h3ve  accomplished 
much. 

Figure  4-20  shows  our  stylized  typical  TV  generation  system 
oase  located  in  main  memory.  All  pointers  illustrated  in  the 
figure  are  absolute  storage  locations.  Associated  with  each 
pointer  is  the  size  of  the  table  that  is  being  oointed  to.  The 
firmware  has  available  to  it  the  base  address  of  the  system  base* 
ROOT.  The  Running  Process  Word  (RPW),  Process  Table  Pointer* 
etc.  are  located  at  fixed*  known  displacements  from  the  ROOT. 
Thus*  the  firmware  may  locate  information  about  any  process  by 
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starting  at  the  ROOT  anc  tracing  through  the  systerr  base.  The 
ith  entry  of  the  Process  Table  points  to  the  Process  Control 

3 1  ock  (PC-8)  of  process  i.  The  PCB  stores  accounting  information. 
CPU  register  values  (for  this  process),  etc.  There  are  also 
pointers  to  a  semaphore  table  and  a  segment  table.  The  semaphore 
table  is  oiscussea.  together  with  the  Running  Process  Word. 
Active  Process  Link  T3ble.  and  G-Ready  Process  below  ir  the 
example  illustrating  F  and  V  operations  on  semaphores.  (See 
Example  6  below. 1  Each  entry  of  the  segment  table  is  a  segment 
tescriptor  which  is  a  pointer  to  that  segment  (if  present  in  main 
.memory),  together  with  status,  and  (ring)  access  information. 
While  we  assume  a  layered  ring  structure,  none  of  the  exsmDles 
presented  below  will  use  it  explicitly  and  so  we  will  represent  a 
segment  descriptor  by  its  location  and  length.  The  handling  of 
access  violations  will  be  very  similar  to  the  bounos  violations 
which  will  be  illustrated  below. 

Rather  than  reproducing  this  complete  system  base  in  the 
examples,  below,  we  will  usually  Just  show  the  segment  table  of 
the  currently  running  process  as  in  Figure  4-21.  This  is  Just 
the  memory  component  of  the  process  map  4.  This,  of  course, 
implies  that  the  name  of  the  currently  running  process.  Pi.  is 
stored  in  the  Running  Process  Word.  RPW. 

In  the  examples,  we  will  also  indicate  the  current  value  of 
the  executing  processor's  instruction  counter.  IC.  as  a  segmented 
address.  This  is  the  IC  value  that*  say.  has  oeen  loaded  from 
the  PCB  when  process  Pi  was  dispatched.  Again,  we  wi I  I  not 
indicate  the  IC's  ring  information  in  the  examples. 
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system,  which  is  a  pointer  to  its  Virtual  Computer  System  Control 
Slock  (VCSCB).  The  VCSCE  has,  say,  four  part**  general  status 
information,  the  virtual  memory  map,  the  virtual  crocessorls) 
map,  and  the  virtual  I/O  mao.  The  examples,  belov.,  will  not 
discuss  the  status  information  or  I/O  map.  The  I/O  discussion 
follows  in  Section  4.7.  Thus,  we  will  normally  show  Just  two 
components  of  the  VCSCB  in  the  drawings.  Most  exsmcles  wil! 
illustrate  a  one  processor  computer  system  in  which  case  the 
orocessor  map  reduces,  largely,  to  the  storage  cf  register 
values.  IThese  are,  of  course,  the  register  values  cf  the 
orocessor  when  it  VM-faulted,  not  of  any  particular  process  being 
run.]  Under  those  conditions,  the  VCSCB  is  sometimes  called, 
col loauial ly,  the  Virtual  Machine  Control  Block  (VMCB). 

The  memory  map  component  of  the  VCSCB  is  illustrated  for 
several  different  memory  maps  in  the  examples.  In  the  exsmoles 
of  a  conventional  relocation  and  bounds  memory  map  R-B,  the 
values  are  shown  stored  in  the  VCSCB.  The  values  are  represented 
as  the  relocation  value  and  the  length  of  the  contiguous 
allocation.  See  Figure  4-23.  In  the  examples  of  a  conventional 
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(fixed  size  block)  pagec  memory  mao*  the  VCSCfi  contains  a  ocinter 
to  the  page  table.  See  Figure  4-24.  The  page  t3ble  may  contain 
status  information  such  as  presence.  us3ga*  ate.  These 
considerations  are  essentially  the  same  as  that  discussed  for  III 
leneration  virtual  computer  systems  C see  Chapter  31.  Ir  our 
examples  we  do  not  illustrate  fictitious  memory  resources  and 
always  show  all  pages  ir  main  memory.  (This*  of  course,  is  not  a 
requirement.!  Thus,  status  information  is  omitted  from  the 
examples. 

The  Virtual  Machine  Identifier  Peqister  (VMIO)  is  also 
illustrated  in  the  examples.  Perhaps,  more  accurately  it  should 
be  called  a  VCSID  bur  the  name  VMID  is  keot  for  historical 
reasons.  The  value  of  the  VMIP  identifies  a  ©articular  VCSC3  (or 
set  of  nested  VCSCB's  ir  the  recursive  case)  anc  defines  the  <rap 
(or  maps)  that  we  have  been  calling  f. 


Example  1  -  Type  I  VCS»  Single  Processor.  P-B  memory  mao 

Ir.  this  example,  we  illustrate  some  simple  instruction 
executions  on  a  virtual  computer  system  implemented  as  a  Type  I 
VCS  with  single  processor  and  relocation  ana  bounos  memory  map. 
Figures  4-25  and  4-26  show  part  of  the  system  base  used  by  the 
V:h  and  part  of  the  system  base  of  VM1,  the  first  level  virtual 
machine  (number  i).  The  dashed  lines  in  Figure  4-25  indicate  the 
segments  of  process  PI  in  the  real  machine,  i.e.  the  VMM.  The 
dashed  lines  in  Figure  4-26  indicate  the  segments  of  process  PI 
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FIGURE  4-26 


Page  143 


in  the  VM.  The  dark  lines  in  Figure  4-26  indicate  the  VM’s 
memory  relocation  and  bounas. 

Figure  4-25  illustrates  the  initiation  of  a  VM  by  the  VMM. 
Since  process  Pi  of  the  real  machine  is  running,  the  RPW  in  the 
system  base  is  PI  [although  this  is  not  shown  explicitly  in  the 
figure]  and  the  processor's  TC  is  11200.  This  segmented  address 
means  segment  1,  word  2G0. 

The  execution  of  instructions  is  traced  in  the  ccmmentsry  to 
the  left  of  Figures  4-25  and  4-26.  We  will  elaborate;  on  the 
commentary  for  this  examole.  The  other  examples  car  be  followed 
without  as  much  elaboration. 

The  instruction  counter  value  must  be  mapped  via  the  process 
-nap  4  from  a  process  name,  1I20P,  into  a  resource  name,  12GG. 
This  is  done  as  a  result  of  the  segment  table  looKuo.  Since 
VMI c=NULL ,  execution  is  on  the  real  machine,  3nd  the  resource 
name,  1210,  is  a  real  resource.  Ccrseauen t I y ,  we  must  fetch  and 
attempt  to  execute  the  instruction  located  at  physical  aedress 
12CC.  This  is  the  LVMID.  A  similar  address  calculation  is 
carried  out  for  the  instruction’s  operand.  The  operand  obtaired 
is  *1*  ana  that  value  is  loaded  into  the  VMIC  register. 
Sonseauent I y,  VMl  is  acTivatec  and  the  information  stored  in  the 
VCSCB,  i.e.  memory  mao,  processor  maos,  etc.,  gets  loaded. 

Figure  4-26  picks  up  the  action  when  the  VvTC  is  1.  Since 
process  Pi  of  VMl  is  now  running,  the  RPW  of  the  VM’s  system  base 
is  PI  [although  this  is  not  explicitly  shown  in  the  figureJ  and 
the  processor’s  IC  is  2I51G. 

The  instruction  counter  value  must  he  maooed  via  the  process 


Therefore*  it  must  oe  mapped  via  the  Vki-jp  f  into  its 


[corresponding  real  equivalent.  Since  the  VNmao's  memory 
coirponert  is  of  form  R-F*  the  real  eauivalent  is  obtainea  by 


& 


E 


checking  the  bounds  and  adding  the  relocation.  9oth  are 
expressed  in*  say*  1000  word  units.  Thus*  fl (2500)=B50C. 
Consequently*  we  must  fetch  and  attempt  to  execute  the 
instruction  locatec  at  ohysical  address  550G.  This  is  the  LOAD 
instruction.  A  similar  address  calculation  is  carried  out  for 
the  instruction's  operand  Isee  steps  (e)-(g)l  ar.d  the  next 
instruction's  address  tstecs  (h)-(k)], 

When  we  try  to  develop  the  address  of  tne  STORE 
instruction's  ooerenfl,  i.e.  U  203u*  we  cause  an  exceptior.  On 
application  of  the  process  rap  4*  we  discover  that  ?G0)  exceeds 
the  bounds  ol  segment  1  in  process  PI  (in  Vhl) .  Thus*  an 


exception  occurs  to,  say,  a  rrivilegsc  layer  of  the  current 
machine.  This  privileged  procedure  may  be  in  crocess  PI  or  it 
might  be  in  some  other  crocess  that  gets  dispatched  as  3  result 
pf  the  exception.  I  See  the  semaphore  discussion,  below.]  In  any 


% 

I 

¥ 


case*  this  event  occurs 
V MM.  The  VMIO  is  sti  I  I 


1/ 


are  handled  automatical! 

In  a  typical  IV 
local  exception  handling 
dynamic  segment  linking. 


without  the  knowledge  or  concern  ot  the 
*1*  as  all  VM  local  exceotions  that  occur 
v . 

generation  system,  similar  exceotiors  and 
might  occur  for  access  violations  or 
These  are  events  that  will  llwely  occur 
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with  higher  frequency  than  in  earlier  III  generation  systems  and 
it  is  important  to  have  them  handled  with  the  minimum  of 
i nterventi on. 

Figure  4-27  is  a  continuation  >f  Example  1.  The  intent  of 
this  extract  is  to  illustrate  possible  referencing  of  the  system 
base  in  the  virtual  machine.  Since  IV  generation  systems  will  be 
unlikely  to  have  an  absolute  addressing  mode*  segmented  addresses 
must  be  used  to  address  the  system  base  even  though  these  entries 
may  use  absolute  adcresses  as  pointers.  [We  have  illustrated  the 
access  to  the  system  base  using  conventional  LOAD  anc  STCPE 
instructions.  This*  cf  course*  assumes  that  the  layered  ring 
nechanism  is  being  used  to  control  access  to  the  system  base.  A 
soecial  instruction  might  be  used  instead.] 

The  VhlQ  contains  *2*  so  V*2  is  currently  in  execution. 
Assume  that  the  active  process*  Fi»  is  running  in  a  privileged 
layer  ard  thus  may  access  the  system  base.  Since  Pi’s  segment 
table  ties  ir.  segment  G*  the  segment  descriptor  ter  segment  2  may 
be  addressed  as  G'102.  Both  steps  (g)  ana  (p.)  which  LOAD  and 
STORE  a  system  base  value,  respectively*  occur  directly  without 
interruption.  This  is  because  the  Vhrrap  f  is  distinct  from  the 
system  case  being  mocified.  In  a  III  generation  software 
implemented  VMmap  f.  noth  steps  (g)  and  (n)  would  have  to  fault 
and  oe  simulated.  As  we  shall  see*  below*  the  V?  [51]  permits 
{g)  to  execute  directly  put  faults  on  (n) . 

(Below*  we  will  discuss  potential  problems  -rising  from  the 


FIGURE  4-27 


I 

P  a  ge  1 5  2  i' 


use  of  an  associative  memory  which  remembers  the  Tost  recent  mao 
comoositiors.  Under  certain  circumstances*  unoracictable  re.utts 


may  occur  in  Doth  the  real  machine  and  in  the  virtual  machine. 3 

Figure  4-28  is  a  further  continuation  of  Example  1.  VM2  is 
currently  running  and  from  f2,  i.e.*  VMCB2*  we  see  that  only  6000 
words  of  memory  are  allocated  to  it.  The  active  process*  P2*  nas 
a  segment  table  which  irdicates  that  9000  words  of  memory  are 
allocated  to  tne  process.  No  validity  check  is  performed  when 
the  system  base  entries  are  stored,  they  are  verified  only  when 
jsed. 

Thus*  in  the  example*  in  step  (e)  application  of  the  process 
map  $(2!30CG)=8GGO  does  not  cause  a  process  exception.  However* 
application  of  the  Vhmac,  f?(8C00)  causes  a  VM-fault  to  the  VMM. 
The  VM-fault  faults  to  the  machine  whose  Vhio  is  formea  by 


.5 


dropping  the  low-orcer  syllable  from  the  current  VMI0.  In  this 
case*  the  fault  is  to  the  null  identifier,  or  the  real  physical 


:h 

\ 


machine.  1 


In  order  to  speed  uo  the  oneration  of  the  crocess  map  4*  the 
neat  machine  will  likely  use  an  associator.  Similarly*  operation 
of  the  virtual  machine  requires  an  associator  to  speed  uc  tne 
composed  map  to*.  CSee  Section  4.5.3  Figure  4-29  sketches 
these  corresponding  associators  and  values  that  would  be  stored 
after  execution  of  the  instructions  of  Figure  4-26.  IPunning  on 
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Associotor —  Reol  Mochine 

After  Figure  4-26  Steps 

(b) 

(e) 


-  Process  P, 


Seg# 

Address 

Validity, etc. 

2000 

4000 

Associotor  —  Virtual  Machine  1 


Process  P. 


After  Figure  4-26  Steps 

(t) 

(f) 


Seg  #■ 

Physico1  Adless 

Validity,  etc 

2 

500'. 

i 

7000 

» 

EXAMPLE  !:  ASSOCiATOR 
FIGURE  4-29 
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the  real  machine  there  voulc  be  no  steDS  (c)  sno  (f),] 

In  both  Coses  we  sssuirc  that  a  cnarge  in  toe  Running  Process 
Wora,  PPW,  causes  the  associatnr  to  os  cleared  of  entries.  In 
the  VM  case,  we  further  assure  that  a  charge  in  VMic  dees 
likewise.  In  any  event,  a  change  by  an  active  process  to  its  rrap 
t  or  certain  otner  parts  of  the  system  base,  e.g.,  PPW,  may  c3use 
unpredictable  results  Jin  the  real  cr  virtual  system). 


Example  2  -  Recursion  Acded  to  Example  1 

In  Example  2*  we  illustrate  the  effect  of  recursion  or  our 
previous  example.  The  two  level-one  virtual  machines  are  the 
same  as  before.  However,  in  this  example,  a  VPM  rurs  on  VMl. 
Pi3ure  4-3,  illustrates  the  VMmap  fragment  in  the  system  base  of 
VMl.  After  VMl  executes  the  ’LVMID  1*  instruction,  tr.e  VMIC  row 
contains  *1.1*.  The  oracKets  on  the  right  sice  of  the  figure 
indicate  the  amount  of  memory  resource  owned  by  toe  real  machire, 
VMl,  and  VMl.i. 

When  we  oick  up  the  action,  process  PI  of  VMi.l  is  executing 
3  seauence  of  instructions  similar  to  that  of  Example  1.  [Note 
that  Pi's  segment  tacle  entries  are  slightly  different  in  this 
example. )  After  applying  the  orocess  mao  4>(2l53i)  we  must  apply 
fl.l  and  then  fl  until  we  finally  man  into  o^vsical  resource 
names.  The  corresponding  associator  is  also  illustrated  in 
Figure  4-30,  Once  again,  segmert  numbers  3’*e  T.aoued  into 
physical  locations.  Tnis  time,  the  entries  represent  fl  o  fl.l  o 
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VCSCB1  VCSC82 

{  3-5  8-6  I 


VMM 
Process  Pj 


ROOT- 


Execution 

(a)  IC  is  21500  ( 

(b)  <f>  (21500)  =  2500 

(c)  fH(2500)  =  3500 

(d)  f,  (3500)= 6500 

(e)  Execute  insh  at  6500 hi 

(f)  <*>(l|20)  =  3020  * 

(g)  fH  (3020)  =  4020 

(h)  f,  (4020  =  7020 

( i )  Contents  of  7020  loaded 

(j)  IC  is  21501 


4100 


0  0-2 
212^7 


Ncte- 

Sfight/y  different 
from  Fig.  4-26 


6500rLQAD~TT 
6501  STORE  11 


Associator- 

clear  on  level  change] 
Seg#  Physical  Loc.  Status 

2  6000 _ 

1  7000  r 


EXAMPLE  2:  EXECUTION  AND  ASSOCIATOR 
FIGURE  4-30 


ihr  t 


% •  Note  that  the  associator 


need  not  be 
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larger  in  the  R -1 


recursive  case  than  in  the  real  rrachine  or  in  the  single  level 
R-B  case. 


While 

we  do 

not 

i 1 lustrate  the 

other  fragments 

f  rot 

Examo  le 

1,  they 

behave 

sim 

i 1 ar 1 y .  The 

process  exceoti 

on  has 

a  iocal 

effect  and 

does 

not  a 

lert  the  VMM 

.  The  system 

b3se 

may  be 

referenced  in  LOAD  or  STORE  instructions  without  causirg  a 
VM-fault.  Finally,  a  VM-fault  by  V-l.i  is  directed  to  VH 1. 


Example  3  -  Similar  to  Example  1  illustrating  VF 

In  the  next  example,  we  contrast  how  the  tferice  Proposal 
[single  level  -  no  recursion]  might  work  for  the  instruction 
seauence  fragment  of  Example  1.  Again,  we  3ssume  two  fixed 
oartitions  of  the  main  memory  for  the  virtual  machines.  Since 
the  VP  does  not  explicitly  provide  the  map  f  tas  the  HV  does]  f 
must  in  effect  be  provided  in  software.  What  the  VF  does 
orovide,  however,  is  the  composed  map  *t  =  f  o  ft  for  all 
processes  of  all  virtual  machines.  Therefore,  the  VCSCB’s  of  the 
VP  do  not  provide  the  memory,  processor  maos  etc.  Instead  each 
VCSCB  points  off  to  the  ROOT  of  a  complete  copy  of  the  translated 
system  base.  Thus,  in  particular,  there  exists  a  path  from  a 
VCSCB  to  its  processes*  .-CB*s  and  ccrrespondi ng  segment  tables. 

In  Figure  4-31,  we  abbreviate  this  structure  by  showing  a 
dotted  line  pointer  from  the  VCSCB  to  segment  table.  The 
contents  of  the  segment  table  for  process  PI  of  virtual  machine  1 
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VMID 


VCSTAB  VCSCBl 
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f\Q  <t> 
Process  Pj. 


(J  3-2 
1  7-1 1 
,  215-2  1 
\  Process  Pg 


Process  P, 


IC  21500 


0  0-2 

1  4-1 

2  2-2 


Execution  of  Instruction  5r 


(o>\IC  is  2 1  500 

(b)  f, o£(2| 500)=  5500 

(c)  Execute  inst.  at  5500 

(d)  f,o  <£(I|20)=7020  7 

(e)  Contents  of  7020  loaded 

( f )  IC  is  2  j  501 

(g)  f,0  4>(2|501)  =  550l  d 

( h)  Execute  instruction  at  5501 

(i)  f i  o  (112000)  =  ^ 

(VP)  VM-fault 


5500  [LOAD  1120 

5501  I  STORE  112000 


7020 


EXAMPLE  3=  VP  EXECUTION 


FIGURE  4-31 
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Exatrple  4  -  Type  I  VCS,  Paged  Memory  Map 

The  next  example*  Figure  4-32,  illustrates  how  a  paged 
memory  map  might  be  added  to  the  VC3CP.  We  assume  that  the  page 
olock  size  is  1000  words  (which  is  also  the  minimum  segment 
unit].  We  do  not  illustrate  fictitious  resources  or  page  faults. 
Although  the  mac  fl  is  somewhat  different  from  Example  1,  its 
application  produces  identical  results.  See  steps  (a)-tl)  for 
confirmation. 

In  the  paged  memory  map,  the  associator  pl3vs  a  significant 
performance  role.  Figure  4-33  illustrates  the  possible  form  of 
this  associator.  Whereas  the  R-B  associator  cid  not  differ 
structurally  from  the  ‘real*  associator  (although  the  addresses 
stored  were  different],  the  paged  associator  has  somewhat 
different  structure  and  will  likely  be  larger  than  the  ethers 
[for  performance  considerations!. 

Note,  once  again,  thai  this  a  sociator  is  different  from 
that  used  in  the  l".?*  360/67  168!  or  the  Honeywell  645  (34,36! 
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ROOT- 


VMID 


VCSTAB  VCSCB1 


Page  Table  VCSCB2  Page  Table; 

'onr  n-»ofn  ; 

■  —  — —  — —  * 

1  6  •••  I  lT 

2  _5_  -  2JOJ  i 

3  4  3j9~~  j 

4_L  J  4  13 

v ^  5  IF* 


Process  Pi 


2|500 


3100 


Execution  of  Instruction  0 

(a)  IC  is  21500 

(b)  4>  ( 2 1 500) 5 2500  b 

(c)  fi  (2500)  =  5500 

(d)  Execute  inst.  at  5500  7 

(e)  ^(1 1 20)  =  4020 

(f)  f,  (4020)  =  7020  8 

(g)  Contents  of  7020  loaded 

(h)  IC  is  2 1  501 

(i)  ^(2|50l)  =  250l 

(j)  fi (2501)  =  5501 

(k)  Execute  inst.  at  5501  « 

(l)  £(lj2000)  =  e  * 

EXCEPTION  TO  PRIVILEGED 
LAYER  OF  CURRENT 
MACHINE 


5500  LOAD  1120 

5501  STORE  112000 


EXAMPLE  4:  EXECUTION  OF  PAGED  V CS 


FIGURE  4-32 


Associates 


After  Figure  4-32  Steps 


(c) 


Seg  #  - 
2  - 


(f) 


1  - 


Page  ^ 

Physical  Location 

Status 

•  0 

5000 

-0 

7000 

1 

EXAMPLE  4:  PAGED  ASSOCIATOR 


FIGURE  4-33 
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•itc.  since  tne  f  and  6  mar  structure  is  different.  The  use  of 
oagino  in  Example  *♦  is  for  memory  rrource  msnsjement  only  ard 
loes  not  contain  anv  of  the  process  structure  normally  fourd  in 
paging  maps.  In  particular,  «=.  •  Figure  4-34  for  ar  illustration 
}f  the  aifference.  Tnus;  while  ■■•ntation,  a  software  visible 
construct,  is  fourd  in  the  4>  (!>*»p,  oaging,  a  resource  bI  location 
mechanism,  is  best  adcec  to  a  system  as  oart  of  in  f-map.  The 
use  of  this  approach  implies  that  complete  operating  systems  will 
oe  runnab I e  without  modification  in  a  paged  environment.  This 
result  should  be  contrasted  with  OS/360  anc  TS5/36r  f 101 .  See 
"Section  4.8,  below. 


•Example  5  -  Example  4  with  Pecursior 

In  Example  6,  Figure  4-35,  we  add  recursion  to  the  paged 
memory  map  of  Example  IV.  As  can  be  seen,  this  example  is  very 
similar  +o  the  recursive  p-8  example  of  Figure  The 
principal  apparent  clfference  is  the  form  of  the  associator. 
However,  because  of  locality  n.^s  iderat i cns ,  =  s  discussed  above 
[in  Section  4.*]  acceotaole  performance  shoul c  result.  Thus,  we 
nave  shown  that  CunliKe  VP]  recursion  does  not  introduce 
additional  complexity,  ever,  in  the  case  M  a  coToiicstad  memory 
nap  such  as  paging. 


C7iiiki.^  .-w 
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FIGURE  4-35 
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Example  f  -  Seiraohores  in  a  Virtual  ^ac^ire 

The  intent  of  this  example  is  tc  further  demonstrate  that 
the  virtual  system  rase  may  be  used  directly  ir  rrcst  instruction 
executions  without  the  intervention  of  the  vmm.  We  will 
illustrate  now  firmw<_.'e  dispatching  and  operations  on  semaphores 
night  work  in  a  virtual  T5chine«  [We  will  assure  the  reader  is 
familiar  with  semaphores  aro  Oijkstra*  P  ard  V  ooerstiorc  [41], 
This  is  crucial  to  thv  operation  of  a  TV  generation  \lu  since  IV 
jererotion  'Joes*  can  be  assumed  to  be  made  up  of  a  number  of 
cooperating  seauential  crocesses  '871. 

The  previous  examples  of  this  section  have  illustrated  that 
oagod  or  other  memory  maos  in  the  Vk C3  introouce  no  sdditioral 
comp  I i cat i or  tnat  cannot  ho  solved  as  in  the  P-3  case.  Thus,  we 
will  illustrate  semaphores  with  an  ®-3  mao. 

First,  we  illustrate  semachc.es  ano  process  discatcnirg  cn 
the  real  machine.  In  Figure  4-36,  frame  (a)  shows  the  Queue  of 
ready  crocesses  pointed  to  bv  the  system  base.  Each  (circle) 
antry  in  the  queue  represents  s  orocess  link  for  some  active 
orocess  in  the  system.  These  links  ccntair  the  o-ocess  name  aro, 
so,  effectively  a  oointer  to  the  corresooncin a  -31.  The  aueue  is 
organized,  cresunably,  in  priority  order,  altnouih  we  do  rot 
illustrate  cr  concern  ourselves  with  the  priority  field. 

Ir  f  r r  e  (o),  tre  first  process  is  r  e  m  o  v  e  c  from  tne  Q  -  p  e  =  g  y 
processes  bv  the  firmware  disoatcrer.  Its  (orocess)  identifier 
is  set  in  the  Punrir?  Process  word,  ppv.  set  us  assume  tnat 
orocess  °1  •‘xecutes  a  p(sem)  ard  the  seirarrore  co^nt  is  ;. 
Ther  n 1  is  clocked  an  c  r j  t  on  the  Queue  of  semsonore  sen.  tne 
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next  orocess*  P2  in  tne-  figure,  is  m?oe  active  [Pr3Te  (cil. 
Jrocess  P2  executes  a  visem]  which  removes  Pi  f nn  ths  semaphore 
tueue  snt  returns  it  to  tne  Q-Peady  Drocesses  [rrame  (d)l. 

All  of  these  man iou I  at i cns  of  the  system  oase  are  normally 
carried  on  entirely  by  tne  firmware  1791.  We  will  shew  that  tnis 
can  be  me  case  in  the  virtual  system  as  well . 

Figure  4-37  provides  a  more  detailed  Dictu'-a  of  tne  system 
oase  illustrating  the  semaphore  tables  and  orocass  links.  Each 
-5CP  points  to  a  semaohore  table  which  defines  tne  semaonores  for 
that  orocess.  If  the  semaphore  exists,  then  tne  semaphore  table 
antry  is  a  pointer  to  the  semarhore  heacer  for  that  semaphore. 
In  Fioore  4-37,  the  semaohore  *sem*  is  callec  seTachora  number  3 
oy  process  Pi  and  semaohore  number  4  by  orocess  0 '  • 

The  semaohore  header  contains  the  count  an:,  if  negative,  a 
pointer  to  the  too  of  tne  semaphore  'aueue .  Tha  cointer  is  given 
as  a  relative  ci so  I acemen t  to  the  oriqin  of  the  Active  Process 
Link  Table.  Subsequent  aueue  entries  are  lirkt  toaether 

V 

similarly.  INote  that  tnis  is  one  possible  v  -  ~  /  «  i  mp  I 

for  semaohores  that  may  oe  usee  in  tnis  il  lustra*'.  .~,Ni  ■  ■  ,  ,Nv, 

! 

macHines.  We  do  not  claim  it  to  ce  the  tos'  ,  -  , 

■  t-  i  *  '' 

argomzat  i  on,  ] 

* 

Figure  a-37  rapreserts  the  system  base  oef.  \  *  • 

♦ 

axecutes  i<e  Pfs^n)  :nstructior.  The  execut;^  jt  Flse-rj  , 

the  firmware  to  acclv  the  t  ai  .  ♦  maps  .as  indicate  in  ♦  h 
commentary.  "inally,  Figure  4-35  shows  r>e  system  -  *  ft  er  the 
=>is?m]  instruction.  Execution  cf  the  sun : ecuer t  Viceml 

instruction  by  orocess  F2  may  be  traced  ir  a  s i t i  | a r  Tanner. 
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8-6 


xecution  of  Instruction 


(o)  IC  is  2  1 504  2- 

(b)  <£(2|504)=2504 

(c)  f  (2504)=  5504 

( d)  Fetch  Inst,  at  5504 

(e)  ^  (sem.O)  =  1616 

(f)  f  (1616)  =4616 

(g)  Decrement  contents  3 - 

of  4616  by  1  VM  ,RooK  See  Value  Above 

( h )  Since  contents  of  t  Pr<*^Tqfale/ 
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EXAMPLE  6*.  SYSTEM  BASE  DETAIL  FOR  SEMAPHORES 

FIGURE  4-37 


Example  7  -  Type  I  ‘jCS,  Multiple  Processors 

This  example  illustrates  the  extension  introduced  by 
multiple  real  and  virtual  processors  in  a  computer  system.  For 
the  purposes  of  the  example  we  assume  that  all  processors  are 
identical  CPU's.  [Clearly  this  work  can  be  extencec  direct  |y  to 
non-t iming-aependent  I/C  processors  as  well  1  15,17,963  .] 

Figure  4-39  shows  the  modification  to  the  VCSC6.  The 
processor  map  now  points  off  to  a  virtual  processor  taole,  each 
entry  of  which  gives  the  (instantaneous)  map  between  virtual  anj 
real  processors.  Also  the  LVMIO  instruction  must  be  expanded  to 
include  a  second  operand  providing  the  processor  number  as  well 
as  the  VCS  number. 

Thus,  when  a  physical  processor,  say,  processor  3  executes 
*  LVMIO  1,1',  the  VM 10  for  the  orocessor  is  set  and  real  processor 
3  is  set  in  the  processor  map  as  com escoca  ing  to  virtual 
processor  1.  Later,  when  virtual  orocessor  i  executes  'SIGNAL  2* 
the  instruction  executes  directly  by  lookirg  up  tr,e  corresc  onding 
real  orocessor  value.  When  'SIGNAL  3*  is  attempted,  since  the 


map  value  is  invalia,  a  pending  bit  is  set.  [A  VM- fault  may  also 
occur  to  allow  the  VMM  to  schedule  virtual  processor  3  ...3 

Note  that  this  example  illustrates  Just  one  way  that  the  f 
map  may  be  introduced  for  multiple  processors.  The  virtual-real 
orocessor  index  is  a  kind  of  page  map  and  the  cisoatching  cf  the 


o 
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EXAMPLE  7:  MULTIPLE  PROCESSOR'S  PROCESSOR  MAP 

FIGURE  4-39 
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resource.  We  can  imagine  other  Kinds  of  processor-  f  maps.  For 
axample,  with  a  relocation-bounds  processor  VM map,  an  induced 
real  processor  number  is  obtained  by  adding  the  relocation  to  the 
virtual  processor  number  after  checking  that  the  bound  is  cot 
exceeded.  One  can  imagine  a  system  organized  around  this  map 
requiring  all  virtual  processors  to  be  assigned  by  start-uo,  i.e. 
no  fictitious  (dynamic)  resources. 


Example  8  -  Type  II  VCS.  Sinqle  Processor 

In  this  examplet  we  give  one  possible  interpretation  of  a  IV 
generation  Type  II  VCS.  To  shew  the  operation,  we  return  to  the 
instruction  seauence  from  Example  1.  Our  system  base  now  has  the 
VCSCB  hanging  off  the  PCB  (of  some  particular  privileged 
process) .  We  have  the  same  memory,  processor,  etc.  components  of 
the  map.  However,  noh  virtual  resource  names  are  mapped  into 
orocess  names.  Thus,  the  memory  map  II  Justrated  takaj  virtual 
memory  addresses  into  segments.  In  our  examp »e,  we  map  addresses 
into  displacements  within  segment  1  of  process  »-l.  Steps  (t)-(d) 
successively  apply  the  Drock-;  -  rac:p  of  i  '  >  ;l  machine,  v , 
the  virtual  machine  map,  f*,  am  r sp  of  the  real 
machine?  4r.  See  Figure  The  a  c>  be  of  t he 
same  form  as  that  used  in  Example  1. 

■» 

As  seen  before,  in  Chapter  ,  'Y  r  ■  e  a  number  of 
advantages  to  running  a  Type  II  VMM .  n  *  ...rst  ion  system, 
a  Type  II  VMM  permits  the  predor  •  iting  system  to  run 
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Execution  of  Instruction 


(o)  IC  is  21500 

(b)  *v  (2 1 500)  =  2500 

(c)  f'  (2500)=  112500 

(d)  (112500)  =5500  7 

(e)  Execute  inst.  at  5500 

(f)  *V(H20)  =  4020  8 

(g)  f'(4020)  =  114020 

(h)  *r (114020)  =7020 

( i )  Contents  of  7020  loaded 

(j)  ICis  21501 

(k)  *V(2I501)  =  2501 

(l)  f,(2501)  =  112501 

(m)  (112501)  =5501 

(n)  Execute  inst.  at  5501  \) 

(o)  *012000)  =  e  * 

EXCEPTION  TO  PRIVILEGED 
LAYER  OF  CURRENT 
MACHINE 


step  seg#  address  validity 

(d)  |2  5000 

(h)  (  1  7000  _ 


EXAMPLE  8  :  TYPE  IE  V  C  S 
FIGURE  4-40 
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without  degradation  due  to  the  VNmap.  furthermore*  it  simnlifies 
the  construction  of  the  VMM  since  the  host  operating  system  takes 
over  the  allocation  of  real  resources.  In  this  example*  virtual 
memory  is  being  mapoed  into  segments  of  a  procass  running  on  the 
real  machine.  Therefore*  the  allocation  of  Tetory  for  these 
segments  and  the  response  to  segment-absent  exceptions  may  be 
handled  by  the  standard  operating  system  tetory  si  location 
procedure. 
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4.7  SOME  PRACTICAL  PROBLEMS  OF  IV  GENERATION  HV 


There  are  a  number  of  practical  problems  that  prevent  the 
hardware  virtualizer*  as  described*  froir  providing  the  complete 
implementation  of  a  IV  Generation  VCS.  Some  of  these 
difficulties*  e.g.»  time,  are  intrinsic  to  virtual  coirputer 
systems.  Other  difficulties*  such  as  the  ao  hoc  unformalized 
tire-dependent  I/O  process*  are  technological  and  may  pass  as  we 
gain  greater  understanding  of  computer  systems  architecture. 
Until  such  time  as  these  problems  are  overcome*  we  can  still  rely 
on  III  generation*  software  VMmap  techniques.  He  will  illustrate 
how  an  interim  solution  can  be  developed  for  I/C. 

The  difficulty  with  I/O  lies  chiefly  with  its  control*  net 
with  memory  or  device  mapping  which  may  be  accomplished  via  the 
VCSC8.  The  interim  solution  we  opt  for  directs  I/O  completion 
interrupts  to  the  real  machine.  Re  maRe  the  Connect  (Start)  I/O 
instruction  (and  all  ofher  I/C  instructions)  absolutely 
or'.vileged  such  that  it  may  only  be  issued  by  the  physical 
machine  3rd  attempted  execution  by  a  VM  causes  s  VM-fault  to  the 
VMM.  as  with  III  -generation  systems*  the  VMM  translates  the 
request  and  issues  ♦he  I/O  request  itself.  On  the  I/O 
termination*  an  interrupt  occurs  to  the  VMM  running  on  the 
absolute  machine.  Regardless  of  the  VM  being  run  at  the  time  of 
the  interrupt*  its  status  may  be  saveo*  and  control  is  returred 


tg  the  virtual  machine  (by  the  physical  machine)  after  interrupt 
processing.  The  VCSC-B  is  expanded  to  include  an  I/O  interrupt 
oending  bit  wh;  •  :s  set  by  the  VMM.  Execution  of  the  LVMID 


and  Snow  1761.  Easicallvt  they  suggest  allowing  the  virtual 
•machine  to  issue  I/O  instructions  subject  to  the  device  msp  in 
the  VCSCO*  I/O  completion  inrerruots  then  come  in  to  the  real 
-nachine  which  oasses  the  information  down  the  VMM  chair  via 


so-called  "spurious  interrupts 
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4.8  II  ANO  TIT  GENERATIONS  REVISITED 
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The  work  of  Chapter  4  has  been  directed  toward  introducing 
the  HV  into  IV  generation  systems.  We  have  assumed  that  IV 
generation  systems  define  an  architecture  wi tn  a  process  mac  $ 
implemented  in  firmware.  We  have  then  expandec  the  design  to 
form  a  new  system  which  incorporates  a  level  invisible  firmware 
VMmap*  as  well. 

We  have  assumed  that  i  is  a  rather  complex  map.  However, 
the  principles  we  have  developed  are  completely  indeoendent  of 
the  complexity  of  6.  Thus*  it  becomes  possible  tc  examine  other 
computer  architectures  in  which  the  maos  f  or  *  may  be  of 
different  form  from  those  considered.  The  important  orooerties 
which  must  be  oreserved,  however*  are  that  t  corresponds  to  the 
software  visible  structure  within  a  level  of  a  virtual  machine 
whereas  f  corresponds  to  the  mao  between  two  levels  of  virtual 
machines.  We  will  consider  two  representative  examples. 

4.8.1  f  =  R-Q  *4>=  i  der,  t  i  ty 

If  we  let  4>=identity  map*  then  we  have  a  computer  design 
without  intra-level  structure.  There  is  orly  one  mode*  i.e. 
"Supervisor  state  is  not  necessary"  [761.  There  is  no  memory 
mapoing  within  a  level.  The  level  structure  looks  very  much  like 
i  first  or  second  generation  design  of  the  sort  found  in  a  CEC 
°0P-8  class  minicomputer. 

The  introduction  of  an  P-B  f-mao  allows  us  to  develop 
virtual  machines  for  this  system.  [As  in  Section  4.5.]  Because 
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of  the  lack  of  structure  in  each  machine  leval.  it  may  be 


tesirable  to  allow  programs  to  communicate  between  levels  by 


causing  a  VM-level  fault  with  a  conventional  bounds  violation 


code*  This  now  defines  s  programming  environment  in  which  the 


VM- level  fault  is  used  to  call  on  supervisory  servicest  much  as 


the  "diagnose"  instruction  in  CP-67  [2.23,65],  In  Figure  4-42, 


the  VMM  running  at  level  n-1  produces  a  VM  at  level  n.  at  level 


n,  we  run  "two-level  operating  system"  OS  fin  the  figure]* 


which  produces  this  enhanced  environment  at  level  n*l.  The 


system  will  work  as  long  as  level  0  to  level  n-1  each  run  the 


VMM.  This  is  true  because  of  VM-recursion  and  level  invisibility 


of  the  f-maps. 


The  "f =R-B»fc=ioent ity"  machine  is  essentially  that  proposed 


oy  Lauer  and  Snow  1761.  They  observe! 


It  is  more  general  than  most  existing  hardware  in  the 
sense  that  it  naturally  encourages  a  hierarchical 
structure  without  imposing  artificial  boundaries  or  an 
omnipotent  suoervisor.  But  it  lacks  some  of  the 
important  features  of  modern  systems,  particularly 
segmented  virtual  memories  and  a  suitable  parameter 
passing  mechanism  ...  1763 


The  introduction  of  intrafevel  structure,  e.g.  process  model  end 


segmentation,  can  best  be  accomolished  using  the  techniques 


developed  earlier  in  Chapter  4.  However,  the  "6=idertity' 


machine  may  still  oe  a  useful  system  to  construct,  oarticularly 


in  conjunction  with  a  line  of  mi ni comouter s .  In  this  cortext. 


ooth  f=R-S  and  f=paging  could  lead  ro  interesting  results. 
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4.8.2  f =oag ing* 0=superv isor/prob I em-state  [no  memory  moping] 

The  work  of  Chapter  4  indicates  how  oaging  could  have  been 
introduced  ,.Uo  an  IEM  System/363  Virtual  Computer  System, 
tnstea-  jf  paging  being  a  part  of  the  visible  intra-level 
structure*  it  should  have  been  introduced  as  cart  of  a  formal 
f-map.  L'-'der  these  circumstances*  there  would  oe  no  need  to  rely 
an  the  techniques  discusseo  in  Chapter  2*  i.e.*  a  "software 

V’mao".  Privileged  instructions  would  be  able  to  execute 
dirict'y  twithaut  the  trap  simulate  dispatch  seauence], 
i  -rs- :evel  exceotions  would  be  directed  tc  the  privileged 
Svliware  at  the  current  level*  recursion  would  be  vastly 
simplified,  and  tne  VMM*  i.e.  CP-67,  would  be  greatly 
streamlined.  Once  again*  a  key  point  is  that  D3ging*  a  mechanism 
for  resource  allocation*  should  properly  be  part  of  the  f-map, 
not  the  O-map. 
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CHAPTER  5. 
CONCLUSION 


5.0  SUMMARY 
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The  thesis  has  develoDed  appropriate  terminology*  an 
empirical  basis,  and  a  model  which  represents  the  execution  of 
processes  on  a  VCS.  The  model  features  two  mapst  C 1 >  a  process 
map  called  t  which  maps  process  names*  e.g.  segments*  semaphores* 
orocess-id’s*  into  resource  names*  e.g.  memory  locations* 
processor  numbers,  and  (2)  a  virtual  machine  mao  (VMmap)  f  which 
■naps  virtual  resource  names  into  real  resource  names.  The 
process  map  is  strictly  an  intra-level  map  expressing  a 
relationship  within  a  virtual  machine;  the  VMmap  is  an 
inter-level  map  expressing  a  relationship  between  (the  resources 
of)  two  adjacent  levels  of  (virtual)  machines.  Thus,  the  action 
of  running  a  process  on  a  VCS  consists  of  running  it  under  the 
composed  map  to*.  Application  of  the  model  has  led  to  a  clear 
interpretation  of  virtual  machine  recursion*  Type  II  virtual 
machines*  and  other  impo'r+ant  properties  of  virtual  machines. 

The  model  also  allows  us  to  understand  the  implementation  of 


to  the  design  of 


hardware  virtualizer 


3 


implementations  of  IV  generation  vCS's.  Tne  design  has  the 
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characteristic  that  all  orocess  exceptions  are  handled  directly 


within  the  executing  VCS  without  software  intervention.  All 
resource  faults  (VM-faults)  by  a  VCS  are  directed  to  its  VKM  on 


p. 


the  real  machine  without  knowledge  of  orocesses  on  the  VCS. 
-aul  +  handling*  invocation  and  execution  of  VCS*s  works  directly 


regardless  of  recursion.  Furthermore,  preliminary  oerformance 
studies  indicate  that  this  design  will  provide  performance  of  the 


The  development  of  the  principles  reported  in  this  thesis 
has  also  pointed  out  a  number  of  areas  for  future  research.  The 

nodel  and  suggested  implementations  of  Chapter  4  have  been 
directed  largely  toward  virtualization  of  a  single  processor 
[without  I/O]  computer  system.  Example  7  of  Section  4.6  sketches 
an  approach  to  multiple  processor  configurations  and  Section  4.7 
suggests  an  interim  solution  to  I/O*  However*  there  is  still 
much  work  to  be  done  in  incorporating  either  of  these  features 
into  the  model  and  the  hardware  virtual izer.  As  Section  4.7 
ooints  out,  progress  in  the  treatment  of  virtual  I/O  is  sotrewhat 
tied  to  progress  in  the  treatment  of  real  I/O.  As  long  as  timing 
dependencies  and  asynchronous  interrupts  exist  in  I/O  systems,  it 

will  be  difficult  to  treat  the  virtualization  of  I/O  in  a 
homogeneous  manner  with  the  rest  of  the  system. 

Another  area  for  further  development  of  architectural 


iround  for  a  number  of  traditiona  I  ly  separata  disciplines  of 
computer  science.  Gne  area*  in  particular*  concerns  the 
development  of  models  which  crovice  an  interior  decor  level 
description  of  the  execution  of  processes  on  3  computer  system. 
The  empirical  work  of  Chapter  3  has  proviced  an  essential 
foundation  upon  which  to  create  a  useful  abstraction  of  program 
execution  in  a  III  generation  VCS. 

The  thesis  has  been  concerned  with  Techanisms  for 

implementing  VCC's*  rather  than  specific  policies  for  aoolyiny  to 

them.  Thus*  we  have  not  discussed  issues  cf  resource  allocation* 

oerformance,  or  throughput  as  applied  to  tne  real  or  virtual 

systems.  This  includes  the  introduction  of  fictitious  resources, 

i.e*  demand  paging*  or  particular  multiplexing  algorithms  to  be 

used  in  a  VNTSS.  It  is  ouite  important  to  develop  models  that 

will  make  possible  reasonable  resource  allocation  decisions.  One 

aspect  worthy  of  study  concerns  treatment  cf  the  process  interval 

timers  and  the  firmware  process  dispatcher.  Should  the  virtual 

machine  dispatcher  be  implemented  as  a  firmware  adjunct  or  will 

software  be.  sufficient?  How  does  this  choice  deoend  on  the 

intended  use  of  the  system,  i.e.  many  or  few  virtual  machines. 

Another  auestior.  concerns  possible  corflicts  and 

counterproductive  algorithms  between  different  levels  of 

virtualization.  For  example*  if  the  t>-irap  is  segmentation  and 

the  f-map  is  paging,  then  memory  compaction  by  the  virtual 

machine  may  be  a  sub-optima,  policy.  AnoTher  examofe  might 

♦ 

involve  "double  spooling"—  once  by  the  virtual  machine  and  once 
by  the  VHk.  Performance  models  and  studies  will  be  a  necessary 


APPENOI*  A 
TUTORIAL  ON  CP-67 


A . 0  Introduction 

In  this  appendix*  we  shall  illustrate  the  typical  software 
implementation  of  virtual  machines  on  third  generation  systems  by 
reviewing  the  mechanisms  used  by  CP-67  19* 32* 54 ,85 ] . 

In  order  to  faithfully  procuce  a  virtual  360,  CP-67  must 
orovide  a  virtual  processor,  virtual  memory,  and  virtual  I/O 
charnels  and  I/O  devices.  The  basic  approach  that  CP-87  adopts 
is  to  handle  high  frequency  events  in  an  automatic  manner  with 
little,  or  no,  overhead,  while  handling  infrequent  events  in  a 
less  efficient  manner.  This  rotion  leads  CP-67  tc  a  rather 
efficient  handling  cf  virtual  processor  en:  virtual  memory. 
Virtual  I/C  events  which  occur  at  a  much  lower  frequency  may  be 
orocessed  with  greater  overhead. 

CP-67  is  able  to  simulate  a  360/67  in  both  of  its  moces  of 
operation.  In  one  mode,  the  360/67  behaves  as  a  350/65*  a 
standard  36P  without  memory  mapping.  In  the  ether  mode*  called 
oy  IRM  "extended  mode",  the  360/67  uses  a  number  of  addi'ioral 
registers  and  instructions.  In  extended  mode,  the  360/67  has  the 
ability  to  enter  relocate  mode.  The  360/57*s  operation  in 
extended  mode,  but  not  relocate  mode,  is  very  similar  tc  the 
operation  of  a  360/65. 


3 

3 


'4 

i 


;:z 

1 


.3 


-sj 


■I 


a 

3 


=3 

£ 

ss 


I 


Page  183 

A.l  CP-67*s  Virtual  Memory 

The  simulation  of  virtual  memory  for  virtual  machines  is 
perhaps  the  easiest  of  CD-67*s  chores*  The  36Q/67  has  a  memory 
relocation  mechanism*  called  DAT  (Dynamic  Address  Translation)* 
which  features  a  segmented-paged  address  scaca  CUE  3 •  The  24  bit 
virtual  address  produced  by  the  360/67  in  relocate  rroce  is 
interpreted  as  a  four  bit  segment  number*  so  eight  bit  page 
number*  and  a  twelve  bit  displacement  witnin  the  page*  Segment 
and  page  tables  ar^  located  in  core. 

CP-67  always  runs  virtual  machines  in  relocate  rode.  Thus, 
the  paging  mechanism  theoretically  permits  virtual  machines  to 
address  a  memory  of  uo  to  2**24  or  16  millior  bytes.  This 
virvual  memory  may  far  exceed  the  amount  of  ohysic3l  memory 
actually  available.  In  practice,  however,  for  reasons  of 
efficiency  virtual  computer  systems  are  normally  configured  to 
nave  a  significantly  smaller  virtual  memory*  usually  lass  than  1 
mil  lion  bytes* 

The  paging  mechanism  may  be  used  directly  as  long  as  the 
virtual  machine*  itself,  does  not  attempt  to  execute  in  (virtual) 
relocate  mode.  If  the  virtual  machine*  e.g.  360/67,  goes  info 
relocate  mode,  it  is  necessary  to  map  the  relocated  addresses  of 
the  virtual  machine  into  their  corresponding  real  addresses. 
This  facility  can  be  provided  with  reasonable  efficiency  via  a 
software  algorithm  as  described  in  Goldberg  [541  and  illustrated 
in  Figure  A-l. 

In  Figure  A-l,  the  oage  map  (C)  provides  the  corresoorderce 
oetween  addresses  in  the  relocated  virtual  memory  (1)  ard  the 
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MAPPING  RELOCATED  VIRTUAL  MEMORY  TO  REAL  MEMORY 
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Thus,  relocate!  virtual 


(nor-relocated)  virtual  memory  (C). 
memory  page  2  is  mapped  into  virtual  memory  cage  7.  Similarly, 
the  cage  mao  (F)  orovides  the  mao  from  virtual  memory  (Q)  tc  real 
memory  (F) ,  Thus,  virtual  page  7  is  mapped  to  real  page  3. 
Since  all  acdresses  must  ultimately  be  mapped  into  real  addresses 
and  the  36C/67  hardware  orovides  only  a  single  level  of  page 
mapping  [this  example  ignores  the  “segmentation”  of  the  360/67J, 
it  is  necessary  to  use  software  to  derive  the  composed  page  map 
(A)  which  maps  directly  from  (P)  to  (F) *  Therefore,  when  the 
virtual  machine  that  is  illustrated  executes  in  (virtual) 
relocate  mode,  relocated  virtual  page  2  is  mapped  into  reel  page 
3. 


A. 2  GP-67 • s  Virtual  I/C 

The  simulation  of  virtual  I/O  channels  and  devices  is 
performed  in  a  number  of  differert  ways  by  CP-67.  Virtual 
machines  may  be  configured  with  I/C  devices  that  are  realized  as 
dedicated,  scooted,  or  shared  devices.  A  dedicated  device  is  a 
device  such  as  a  tape  Irive  or  an  entire  disk  pack  which  may  be 
used  by  only  or.e  virtual  machire  ai  a  time.  Consecvjent  ly ,  tape 
drives  arv;  attached  to  some  virtual  machine  for  a  given  duration. 
The  virtual  device  adcress  need  not  be  the  same  as  its  physical 
address.  A  corresponcence  between  the  two  is  established  at  the 
time  that  the  device  is  attached. 

A  shared  I/O  device  is  a  physical  device  which  may  be 
simultaneously  part  of  the  configuration  of  a  number  of  different 
virtual  machines.  A  shared  I/O  device  is  divided  among  each  of 
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the  oifferent  owners  in  such  a  way  that  each  virtual  machine 
oelieves  that  it  alone  owns  the  device*  but  th3t  the  virtual 
device  is  smaller  than  the  actual  physical  device.  An  example  of 
a  shared  device  is  a  disk  pack  that  has  been  dividec  among 
several  virtual  machines.  These  so-called  aicizdiSlSS  behave 
exactly  like  the  ordinary  IBM  2311  disks  exceot  that  tney  have 
fewer  cylinders.  Another  example  of  a  shared  I/C-  device  is  a 
transmission  control  unit*  such  as  the  IBM  2702*  in  which  each 
virtual  machine  may  have  only  a  small  number  of  the  transmission 
lines  into  the  device.  Such  sharing  artifacts*  like  the 


mini-dis:.*  are  introduced  for  cost  or 


‘ef  f  iciency* 


considerations.  They  improve  the  sharing  of  large  resources  that 
might  otherwise  be  ineffectively  utilized. 

Spooling  is  the  third  methoc  which  CP-67  uses  to  simulate 
I/C  devices.  Each  virtual  machine  has  attached  to  it,  3  virtual 
card  reader,  virtual  card  bunch*  and  virtual  line  printer. 
■Rather  than  provide  a  separate  real  device  for  each  virtual 
machine*  CP-67  utilizes  buffered  disk  files  to  simulate  these 
devices.  Or  output,  data  directed  to  the  virtual  printer  or 
ounch  are  written  on  a  soecial  disk  file.  When  the  real  physical 
device  becomes  available,  the  temporary  file  is  processed  all  in 
one  piece*  Similarly*  card  input  is  first  read  in  by  CP-67  and 
written  on  a  disk  file.  Card  images  are  then  provided  or  each 
virtual  I/O  read  reouest. 

Dedicated  I/O  device  simulation  is  the  most  general  approach 
vised  by  CP-67  and  may  be  used  for  devices  which  are  unsupported, 
or  supported  in  other  ways.  Thus*  alien  devices*  such  as  the  IBM 
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725C  graphics  console  C831  may  be  connectea  providing  that  the 
virtual  machine  contains  the  appropriate  I/O  Handling  support. 
CP-67  need  or.  I  y  provide  very  simple  interrupt  handling.  Oevices* 
such  as  the  reader,  punch,  or  printer,  which  are  normally  spooled 
Icvices,  may  be  attached  to  a  single  virtual  machine  if  desired. 
Oevices,  such  as  disk  packs  or  telecommunications  control  units, 
which  are  normally  supported  by  CP-67  through  sharing,  may  be 
defeated  to  a  single  virtual  machire  on  a  whole  device  basis. 
Thus,  CP-67*s  handing  of  I/O  devices  is  suite  flexible  and 
allows  for  configurations  which  are  somewhat  different  from  the 
actual  resources  available. 

Since  a  virtual  device  aedress  may  differ  from  its 
corresponding  physical  device  address  (that  is  used  to  realize 
it)  it  is  necessary  to  perform  some  kind  of  device  mapping  before 
ah  I/O  transfer  may  take  place.  The  device  mapping  is 
illustrated  in  Figure  fl-2.  A  virtual  device  is  located  on  seme 


oarticular  virtual  channel  and  virtual  control  unit.  Once  tt e 
ievice  correspondence  is  made  with  the  real  I/O  device,  its 
oarticular  real  control  unit  and  channel  may  be  identified.  I/O 
transmission  reauests,  CCW  (Channel  Command  Wore)  chains,  must  be 
translated  to  reflect  the  corresponding  real  I/C  device  involved 
(as  well  as  the  physical  location  in  core  memory).  The  CCW 
translation  is  done  and  the  I/O  request  is  placed  on  the  queue 
for  the  real  channel.  When  a  completion  interrupt  ccmes  in  on 
the  real  channel,  this  information  is  posted  b3Ck  to  the 
corresponding  virtual  channel. 

Because  of  the  recessity  of  translating  CCW  chains  before 
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I/O  initiation,  CP-67  introduces  the  adaitional  restriction  that 
no  charnel  program  is  changed  by  the  CPU  or  the  channel  during 
the  interval  between  execution  of  the  START  I/O  instruction  and 
the  charnel  end  interrucl  except  that  oerformed  by  the  OS  indexed 
seauential  access  method  (ISAM).  The  restriction  arises  because 
CP-67,  naving  transletec  the  channel  program*  will  have  no  way  of 
Knowing  that  it  has  been  changed  and  should  be  translated  again 
[81. 

The  actual  mechanism  that  allows  CP-67  to  gain  control 
before  a  virtual  machine  can  start  up  I/O  will  be  explained  below 
in  the  section  on  the  virtual  processor. 
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A. 3  CP-67  *  s  Virtual  Processor 

C°-67  simulates  a  virtual  processor*  e.g.  360,  by  using  the 
virtual  machine  mechanism  that  was  suggested  in  Chapter  1  ard 
Chapter  2.  Since  the  virtual  processor  and  real  host  are  similar 
(or  identical),  the  virtual  processor’s  opcodes  may  be  executed 
directly  on  the  host  machine.  Thus,  an  instruction- 
oy-instruction  interpretation  is  not  reauired.  H, never,  it  is 
necessary  to  prevent  certain  instructions  from  executing  directly 
on  the  host  machine.  These  instructions  if  executed  directly  by 
a  virtual  machine  can  cause  serious  difficulties  in 
interpretation  or  control.  Any  instruction  that  will  rot  be 
oeriritted  to  execute  directly  on  the  host  machine  is  termed  a 
5SDSjLLU£  instruction. 

The  approach  that  CP-67  adopts  to  prohibit  the  direct 
exec..*f.on  of  sensitive  instructions  is  to  run  the  virtual  machine 
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always  in  (physical)  proolerr  state  cn  the  host  machine.  Recall 
that  va  real  363  has  two  modes  of  operation:  (1)  supervisor 
state,  in  which  ol !  instructions  are  permittee  to  execute,  and 
(2)  problem  state,  in  which  only  s  subset  of  the  instructions  are 
allowed  direct  execution.  The  attempt  to  execute  a  privileged 
instruction  (one  which  nay  only  be  executea  in  supervisor  state) 
while  in  problem  state  causes  a  program  interrupt.  i.e.  trap.  By 
running  a  virtual  machine  only  in  problem  state,  CP-67 
effectively  prevents  the  direct  execution  of  all  privileged 
instructions.  If  the  sensitive  instructions  are  privileged  then 
CP-67  has  succeeded  in  creventing  their  direct  execution. 

CP-67  maintains  a  number  of  tables  in  its  suoervisorv  area. 
Among  them  are  copies  of  all  working  registers  that  the  virtual 
machine  believes  that  it  has.  The  registers  include  a  virtual 
orogram  status  wore  FSh.  This  register  indicates  the  state  that 
the  virtual  machine  believes  that  it  is  in.  If  the  virtual 
machine  is  in  (virtual)  problem  state  sne  a  privileged 
instruction  is  attempted,  CP-67  reflects  this  fact  back  to  the 
virtual  machine.  The  virtual  machine  is  cut  into  (virtual) 
supervisor  state  so  that  it  may  process  the  program  interrupt 
itself.  The  real  host  machine  still  remains  in  problem  state 
while  the  virtual  machine  is  running.  If  the  virtual  machine  is 
in  (virtual)  supervisor  state  and  a  privileged  instruction  is 
attempted,  CP-67  recognizes  that  this  woulc  be  a  valid 
instruction  on  the  real  machine  ard  simulates  the  effect  cf  the 
instruction. 

Some  of  the  privileged  instructions  of  the  360  which  CP-67 


Control  (LMC) ♦  Store  Multiple  Control  (STMC),  and  Load  Real 
Address  (LPA).  Since  the  Start  I/O  (SIO)  instruction  is 
orivileged,  the  attempt,  by  3  virtual  machine,  to  perform  I/O  is 
immediately  recognized.  It  is  at  this  point  that  CP-67  may 
oerform  the  mapping  between  virtual  and  real  I/O  devices  (as 
lescribed  above)  and  then  initiate  the  corresponding  I/O 
operation  itself.  The  storage  protection  instructions  are 
orivileged  and  this  allows  CP-67  to  Keep  a  copy  cf  a  virtual 
machine's  protection  Keys  and  use  them  when  the  vintual  machine 
is  dispatched.  LiKewise,  the  instnuctions  which  start  up  the 
relocation  system  are  orivileged  and  so  virtual  relocation  fray  be 
napped  like  virtual  I/C  [5AJ. 

Thus,  CP-67  simulates  a  virtual  360  processor  through  a 


combination  of  hardware  and  software.  It  insulates  the  (real) 
host  machine  from  actions  of  the  virtual  machine  by  always 
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CASE  STUOIES  OF  SOME  III  GENERATION 


M  ACM  T NE S 


T.O  Introduction 

In  this  appendix*  we  provide  a  number  of  ewsaoles  of  third 
generation  machines  wnich  are  and  which  are  not  vi  rt,j  a  I  i  zap  I  e. 
For  the  machines  which  are  not*  we  indicate  which  of  the 

jmpirical  rules  ot  Section  3.2  are  violated.  Tnese  results  *re 
summarized  in  Figure  o-2.  This  set  of  examples  is  not  meant  to 

•oe  an  exhaustive  survey.  Rather*  it  is  inteued  to  give  the 
reader  representative  examples  of  problems  that  or.e  jncounters 
with  c onteirporory  macnines.  Inoeed*  it  snbee-s  that  whether  or 
not  a  particular  nost  sjoports  virtual  machines  is  merely  a 


natter  of  chance. 

This  is  due  to 

the  f  id  roc t 

non;,  of 

tne 

naci.ines  cited  in 

this  section  were 

ever  desi.jne 

d  with 

the 

thought  of  supporting  virtu;.  I  macnines. 

d.l  IHM  36  /o 7 

CP-67  has  been  in  existence  for  several  y»a~s  now*  so  there 
can  be  little  oouot  concerning  the  36i/67’s  ability  to  support 
virtual  machines.  Tne  initial  CP-67  ran  on  a  3oV67  out  produced 
only  virtual  standard  36C*s»  e.g.  36. /zr>.  Under  these 
conditions*  CP-67  was  not  se I f -v ir tua I i zing  and  so  not  strictly 
under  consideration  here.  Tne  extensior  of  CP-67  to  support  a 
virtual  360/L7  maKes  it  se I f-virtusl izing  and  jppropriate  for 


study  here 
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CP-o7  is  a  Type  I  VCS  which  is  a  timesharing  system*  i.e. 
VMTSS.  It  proouces  virtual  machines  using  tne  oasic  mechanism 
iescribed  m  A-pendix  A.  The  360  has  two  rurcwir ?  modes  of 
doeration  arc  so  it  is  aoorooriate  to  apply  the  rules  developed 
in  section  3.2.  Pule  1  is  satisfied  sm:e  tne  instruction 
execution  is  identical  for  non-pr i v i I eged  instructions  in  Potn 
supervisor  and  problem  siate.  This  is  a  cnar acteristic  of  al? 
i&.'s*  not  Just  the  36o/o7.  Rule  2  is  satisfied  since  tne  36. /t? 
TAT  relocation  system  may  be  used  to  isolate  thj  Virtual  machine 
from  the  VMN.  Rule  3  is  satisfied  since  sensitive  instructions 
cause  recoverable  traos.  Ir>  particular*  ir  the  3-3D/o7,  all  of 
the  instructions  which  can  cause  problems  are  orivileged.  Thus* 
Rule  5»  is  satisfied  by  LPbk*  SSM,  SVG*  DIAG,  L 1C*  bTRC*  LRA  las 
explained  in  Appendix  AT.  Rule  3b  is  satisfied  Decause  tne 
relocation  system  makes  tne  hardware's  reserved  core  locations 
inaccessible  to  the  virtual  machine.  Again*  the  instructions 
which  reference  tne  reserved  registers*  such  rs  the  PSW  or 
Control  Registers  are  privileged.  Pule  3c  is  satisfied  oecause 
tne  appropriate  sensitive  instructors  3r »  privileged. 
Furthermore*  the  use  of  tne  memory  relocation  system  in 
•'managing”  the  virtual  machine  does  not  introduce  idditional 
unsolvable  complications.  Thus*  relocation  of  victual  memory  may 
occur  without  error.  Any  traps  that  occur  in  reconciling  virtual 
addresses  with  real  addresses  or  even  relocated  virtual  addresses 
with  real  addresses  do  not  cause  unrecoverable  ?rrors. 

Two  points  are  of  considerable  interest  in  this  discussion. 
First*  the  360/67  prechecks  instructions  whicn  ray  cross  page 
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ooundaries.  Thus*  instructions  which  may  cause  difficulty,  such 
is  the  Aad  Decimal  instruction,  are  suppressed  oefore  execution 
rather  than  terminated  during  it  t5**J.  The  seejni  point  is  that, 
in  reconciling  relocated  virtual  addresses  with  real  iddressos, 
the  OAT  itiecnanisni  may  oe  set  to  trap  until  this  manory  napping  is 
correctly  established.  Rule  3 a  is  satisfied  because  tne  various 
instructions  which  reference  I/C  are  privileged. 

Thus,  not  surori sing  I y ,  the  3bQ/£?  satisfies  ill  of  the 
empirical  hardware  requirements  stated  in  Sectian  3.2. 


1.2  IGM  36U/P5 

In  this  section  we  discuss  the  problem  of  i  mp  I  e nent in g  a 
CP— t  3  system.  CP— 35  is  a  VCS  which  runs  on  a  3iQ/65  to  produce 
virtual  3bi/65*s. 

As  was  mentioneJ  above,  a  standard  3SU,  e.g.  3cu/cb, 
satisfies  VCS  naroware  requirements  one  ami  two,  with  regard  to 
Rule  2,  the  36U/65  has  no  relocation  system  so  tnit  if  must  rely 
on  its  storage  protection  system  to  protect  the  VMM.  It  also 
must  use  the-  protection  system  to  prevent  instructions  from 
referencing  the  special  system-wide  registers  (Rule  3o)  stores  in 
low  core  since,  in  general,  they  may  be  ,cctssei  without 
privileged  instructions.  Rules  3a»  c,  u  are  satisfied  because 
the  relevant  instructions  are  oriviieged.  Thus  it  is  sufficient 
to  see  if  the  storage  protection  system  is  ouesuite  to  satisfy 
conditions  c  and  3b. 

Since  we  are  using  the  protection  system  to  protect  queries 
as  well  as  alterations,  we  are  led  to  tne  conclusion  that  we  need 
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the  36_*s  five  bit  Key  store-and-f etch  orotecT  system,  With  this 
fectu*-e  each  2F  byt*'  half-page  is  assigned  a  starige  Key  via  the 
orivilegaa  SSK  instruct  ion.  Four  of  t  no  oits  form  j  store 
protect  Key  and  the  fifth  bit  (nay  be  set  ta  indicite  fetch 
protect  as  well.  If  the  four  bit  key  in  the  3S^  corresponds  to 
the  four  bit  Key  assigned  to  a  2K  block*  then  trie  task  is 
permitted  to  fetcn  or  store  into  the  olock.  If  the  ktys  co  not 
natch  then  the  task  n3s  only  fetch  access  to  tna  oIock  unless  the 
fetch  protect  oit  is  also  set.  If  the  key  in  the  PSW  is  z^ro 
then  the  task  nas  fetch  ano  store  access  to  any  pIock  of  storage 
regardless  of  the  Key  storea  with  it. 

Thus*  for  CP-63*  3  zero  storage  protection  Key  in  tne  PSW  is 
analogous  to  being  in  supervisor  state.  It  gives  the  holder  more 
power  than  he  may  be  able  to  use  wisely.  It  is  too  much  power  to 
jive  to  a  virtual  machine  and  so  the  virtual  machine’s  <ey  in  the 
3SW  must  be  some  other  value.  Only  tne  supervisor  may  actually 
run  with  storage  key  zerj.  The  virtual  machine  runs  with  storage 
key  "virtual  zero"  in  which  the  suDerviso-  oerforms  a  Key 
translation  for  the  virtual  machine. 

While  this  scheme  adequately  limits  virtu*!  memory  and 
protects  the  supervisor*  the  dedicated  locations  in  low  core  (tne 
first  2<  block)  cause  it  trouble.  Since  tnera  is  no  relocation 
scheme  on  a  standard  36.  anc  the  machine  can  generate  and  store 
absolute  addresses*  the  virtual  machine  runs  unr  ■»  l  oc  at  a  3.  Tnus* 
the  virtual  machine's  addresses  . -2K  correspjnt  to  the  real 
addresses  -2K.  In  order  to  prevent  the  victual  machine  from 
accessing  the  interval  Timer*  new  PSH’s  etc.*  tie  first  2K  block 
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nust  oe  fetch  and  store  protected  from  all  virtue!  machines.  In 
order-  to  pi-rservo  the  notion  of  £  virtual  machine,  there  must  be 
a  set  of  corresponding  virtual  special  registers  stored  somewnere 
else.  These  may  be  <eot  in  upper  core.  Attempts  to  access 
registers  will  be  trapped  by  the  storage  protection  macnanism  and 
the  supervisor  may  simulate  the  etfect.  This  metnoci  of  using  the 
protection  Keys  to  prevent  the  virtual  macnine  fro?,  executing 
sensitive  instructions  works.  Unf ortunate  ly  si  ice  th  i  protection 
system  on  the  36.*  involves  2K  blocks  of  storoje,  tna  care  above 
byte  location  128  (decimal)  in  block  Zero  is  protected 
unnecessarily.  As  a  resjlt  any  instruction  in  core  location  less 
tnar  cK  or  any  instruction  accessing  data  i  r-  core  locition  Isas 
than  2K ♦  will  be  trapped  automatically.  This  will  occur  for  all 
instructions,  not  Just  for  the  sensitive  ones.  Consequently  in 
order  to  preserve  tne  virtual  machine  notion,  toe  supervisor  must 
simulate  eacn  such  instruction.  This  can  C3use  a  significant 
loss  ir  efficiency. 

Actually,  tnere  is  at  least  one  somewhat  jnlikely  condition 
jnder  which  such  3n  instruction  may  not  be  correctly  simulated. 
If  a  stor^ge-stor«ge  type  instruction,  like  AGO  )ECIf*AL  is 
interrupted  during  execution,  due  to  a  protection  exception,  the 
instruction  is  terminated  and  partial  results  may  be  lost. 

3.3  HIT  AC  !54  tic 

The  HITAG  is  the  Japanese  equivalent  of  the  Spectra 
t o/A5,  As  such,  it  is  very  similar  to  a  standard  36  ,  e.g. 
J6v/b 6,  and  so,  the  preceding  discussion  is  comoletely 
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jpplicsble.  It  satisfies  the  empirical  n^rdwa-'e  requirements  in 
the  same  way  that  the  36J/65  did.  above. 

1.4  I B H  3l.. /<55> 


|  The  IRM  Zh./Q'i  is  o  standard  member  of  the  io  family.  Its 

St 

|  interior  decor  is  similar  to  the  other  stanlird  noiels.  e.g. 

I 

|  36v/6G.  Its  principal  structural  cnange  from  tie  other  nolels  is 

h 

|  the  introduction  of  a  small  buffer  memory  cilleo  trie  cache. 


3ince  tne  cacne  is  invisible  at  the  machine  arcnitectjre  level, 
i.e.  not  oart  of  the  interior  aecor.  it  need  not  oe  virtualized, 
and  need  not  even  enter  into  virtual  machine  consi  terati  ons. 
Since  any  standard  3G0  may  be  virtualized,  tne  3Sj/«5  may  oe. 


l.‘i  G«=ta  General  NCVA 


Trie  0?tj 

General  MOV A  and 

SUPERNOVA 

or*  sixteen 

oit 

minicomputer  s. 

Recent  work  by  the 

author 

and 

L .  Dicimsn  has 

i  ed 

to  the  d« si •...»■  of 

a  VCS  for  the 

NOVA. 

The 

virtual  NOVA 

is 

facilitated  oy  the  memory  allocation  and  protection  ootior.  tnat 
is  avai  I  it  I  *  •  This  option  introduces  p3giry  -*'\S  two  orocessor 
Bodes —  supervisor  and  user.  Since  all  instructions  whicn 
manipulate  state  information.  e.g.  the  page  table  registers,  are 
"I/G  instruct  ions**  ohd  all  I/O  instructions  are  privileged*  Rules 
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hardware  mooes  of  operation,  ._•  store  (but  not  fetch)  orotection 
system*  and  r.o  relocation  system.  The  rrac-une  cannot  be 
virtualized  for  a  number  of  different  reasons. 


Ru  I  e 


is  generally  violated.  Seme  of  th  i  sensitive 


instructions  are  trapped*  either  as  a  result  of  the  .privilege 
nechanism  or  the  protection  mecnanism.  However*  for  those 
i nstruc t i ons*  the  trap  is  oeftrred  one  full  instruction 
execution.  Thus*  the  state  of  the  machine  will  o  i  changed  before 
the  supervisor  gains  control.  In  fact*  it  is  even  oossiole  for 
the  instruction  following  the  sensitive  instruction  to  store  into 
that  instruction  oefore  the  trap  has  occurred.  tow,  the  VMM  nas 
no  hope  of  simulating  the  instruction. 

Pule  3.-  is  violated  in  a  manner  similr-r  to  tvj  P.3P-IG.  iSee 
oelow.]  The  instruction  used  to  return  from  sjoarvisor  state  to 
oroblem  state  is  ERM  (Enter  Restricted  Mode).  Unfortunately* 
this  instruction  is  not  privileged*  Thus*  its  execution  in 
virtual  supervisor  state  will  rot  cause  a  trap.  i  virtjol  state 
change  may  occur  without  the  Knowledge  of  the  VMM, 

Rule  3b  is  violated  because  of  the  lacK  of  a  relocation  or 
fetch  protect  system.  Tnis  difficulty  by  itself  nee J  not  he 
fatal  since  the  introduction  of  a  suitable  restriction  can  still 
oroauce  a  useful  virtual  machine  (as  in  Fuch  i  t-*9)>.  However* 
Pule  3b  is  violated  in  another  more  severe  manner  with  the  TMA 
(Interchange  Memory  and  Accumulator)  instruction.  If  IMA  is 
jttemoted  into  an  area  tnat  is  protected*  the  accumulator  will  be 
loaded  before  the  orotection  violation  is  signalled. 
Consequently*  it  is  impossible  to  recover  the  original  contents 
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of  the  accumulator  and  simulation  is  not  possible. 


•3.7  GE  635 »  d55  (Honeywell  6,<-.) 


The  Gc  635  cannot  be  virtualized  because  >f  violations  of 


Rules  3a  ano  3c.  The  LOT  (Load  Timer)  ana  L.3AR  ( I  o.ad  Base 


Address  Register)  instructions  are  sensitive  instructions  that 


null  execute  rather  than  trap  when  issued  in  problem  state  (si  we 


node).  The  G£  655,  a  successor  to  tne  635  attamats  to  correct 


this  error  by  providing  a  trap  on  LOT  or  L3AR  in  problem  state. 


However,  there  are  still  difficulties  with  the  nachine  oecause  of 


visibility  to  the  physical  master/slave  node  via  the 


non-pri vi I eged  store  indicators  instruction. 


3.8  iultiaota  Model  A  (3EL) 


Tne  Multioata  Modal  A  is  a  16  bit  mini-comouter  witn  3 


hardware  implemented  aaging  system.  A  virtual  sachine  system  is 


impossible  since  Rule  3  is  completely  violated.  All  privileged 


instructions  execute  os  NOP  when  in  problem  stat*.  These 


instructions  include  tne  status  switching  instructions  and  I/O 


instructions. 


3.9  XOS  94  ( 


Tne  XOS  94*  cannot  oe  virtualized  because  of  «  fundamental 


violation  of  Rule  1.  Bit  0  of  each  instruction  word  indicates 


whether  or  not  the  memory  mapping  is  to  oa  on  for  that 


instruction.  The  bit  has  effect  only  in  supervisor  state  out  is 


ignored  in  problem  state  where  it  is  assumed  that  all  addresses 


m 
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ire  mapped.  Thjs,  running  the  virtual  supervisor  state  as 
physical  proolem  state  .does  not  provide  3  mechanism  to 
distinguish  between  mapped  and  unmapped  addresses. 

3.1.  DEC  PDP-1 j 

The  OLC  PDP-10  may  not  be  virtual i2ed  oecause  it  violates 
I  e  da.  The  JRSTF  (Jump  and  Restore  Flags)  or  “JRST  1,"  (Jump 


a 


3 

■sa 

I 

s' 


s 

I 

<<’ 

lx 
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to  User  Program)  instructions  are  used  to  return  from  supervisor 
state  to  problem  state.  When  executed  in  (anysical)  oroblem 
state,  the  instructions  do  not  trap.  Thus,  it  is  impossible  to 
tetect  o  virtual  state  cnange. 


Anotner  violation  of  Rule  3a  concerns  the  iccessi oi  I  i ty  of 
the  “flags”  wmcn  indicate  what  physical  mode  the  machine  is  in. 
Thus,  a  program  in  virtual  supervisor  state  mignt  discover  from 
the  flags  tnat  it  is  ir.  (physical)  orobies  state  and  become 
“confused”. 

As  with  several  other  machines  wnich  violate  Rules  1  or  3a, 
it  is  oossiole  to  construct  a  hybrid  virtual  nacnine  on  the 
30P-1  .  ISee  Section  3.3* J  The  HVM  is  possible  since  31  attempt 
oy  a  virtual  macnine  to  emer  supervisor  state  traps  to  tne  VMM. 

Tne  means  of  entering  suoervisor  state  are  vi 3  a  (privileged) 
JUO.  Thus,  an  HVM,  Type  I  or  Type  TI,  is  possiol j  on  tne  POP-iO. 
The  organization  of  a  Typo  II  HVM  is  discussed  in  Appendix  C. 

3.11  B8N  TLNEX 

TtNtX  12. J  is  a  POP-1*,  that  has  been  modified  to  include 
nardw3re  paging.  This  modification  has  not  changed  tne  situation 


a> 
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APPENDIX  C 

CASE  STUDIES  OF  THIRD  GENERATION  TYPE  II  SYSTEMS 


C.O  Introduction 

In  this  appendix,  we  consider  examples  of  several  different 
operating  systems  which  co  and  do  not  support  Type  II  (extended 
machine  host1  VMM's.  For  the  machines  which  are  Type  II 
v irrua I izab le ,  we  indicate  the  modifications  that  had  to  be  made 
to  support  the  empirical  software  reauirements.  For  the  systems 
which  are  not,  we  indicate  the  empirical  rules  which  are 
violated.  These  results  are  summarized  in  Figure  3-8.  The 
PQF-l J  did  not  satisfy  the  harcware  reauirements  but  the  ITS 
operating  system  satisfies  the  software  rules.  Ccnsecuent I y,  we 
are  able  to  exhibit  a  Type  II  hybrid  virtual  machine  for  this 
system.  As  with  the  hardware  examples  of  Aooendix  B»  the 
operating  systems  investigated  in  this  section  were  not  designed 
with  the  thought  of  supporting  virtual  machines. 


C.l  HITAC  8400 

The  HITAC  8400  149]  has  a  Type  II  virtual  machine  which  runs 
under  the  TCS/TOOS  operating  system.  The  HITAC  8400  is  (more  or 
less)  Type  I  virtual izao I e  Isee  Appendix  P).  TOS/TDCS  satisfies 
software  rule  SI  but  does  not  satisfy  S2  or  S3.  Consequently, 
modifications  were  race  to  provide  protection  on  3n  ad  hoc  basis. 
Other  ad  hoc  solutions  were  introcuced  to  eliminate  some  cf  the 
violations  of  Rule  S3.  For  example) 
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A  supervisor  call  instruction  (SVC)  is  not  a  orivileged 
instruction  but  it  causes  interruots  changing  the 
current  program  state  to  P3»  As  TOS/TOCS  had  no 
set-exit  (unction  for  this  instruction*  we  used  an 
illegal  code  instruction  instead  of  a  SVC  instruction 
to  permit  simulation  through  a  (sic)  illegal  code  trap 
[491. 

The  authors  indicate  some  of  the  Type  II  requirements. 

The  supporting  operating  system  must  provide  a  "set 
exit**  or  "ON1*  function  for  these  types  of  traos  caused 
by  hardware.  Aside  from  orivileged  instructions*  the 
so-called  SVC  (supervisor  call)  instruction  must  also 
be  “set  exited**  and  simulated.  The  SVC  normally 
communicates  with  the  current  supervisor  but  now  it 
must  be  forced  to  direct  to  the  suoervisor  simulated 
(491. 


Recognizing  the  inelegance  of  their  system*  the  authors  orooose 


some  improvements* 


As  for  the  T00/T00S*  the  interval  timer  should  have 
finer  precision  and  set-exit  functions  for  address 
errors  and  SVC  should  be  provided  as  standard  features. 
In  addition*  "set  storage  key*"  "set  time,**  and  "set 
CAW*  functions  should  be  prepared  [491. 


C.2  360/67  UMMPS 

A  Type  II  virtual  machine  has  been  constructed  on  the  Uht'PS 
system  for  the  IBM  363/67  [4*6C1.  Since  UMM°S  did  not  satisfy 
Rules  S2  and  S3*  it  was  modified  to  include  several  new 
orimitives.  The  most  ircoortant  of  these  is  cal  lea  SVC  SWPTRA.  A 
very  brief  description  of  the  virtual  machine  monitor  (called 
"the  monitor”*  below)  and  the  action  of  SVC  SWPTRA  is  given  in  an 
informal  user's  guide. 


When  you  run  the  monitor*  you  tell  it  (by  way  of 
the  VARY  command)  which  virtual  devices  are  attached  to 
which  real  devices.  You  then  issue  tne  IPL  command  and 
the  monitor  fakes  the  IPL  sequence.  After  this*  you 
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communicate  with  the  virtual  system  by  way  of  commands 
which  make  available  the  360  console  functions  (display 
snd  alter  memory,  PSk  restart*  exterral  interrupt* 
console  request*  etc.) 

The  monitor  acouires  space  for*  and  loads  the 
virtual  system  into  segment  3.  The  reason  for  using  a 
new  segment  is  this*  when  the  virtual  machine  is 
running*  it  must  have  addresses  0-236K  (or  0-512K  or 
whatever).  This  is  accomplished  by  SVC  SWPTPA*  which 
swaps  the  segment  table  entries  for  segment  0  and 
segment  3*  detaches  segments  0  ana  1*  and  4+,  and 
transfers  into  the  virtual  system.  This  takes  care  of 
the  addressing  problem. 

Now  the  virtual  machine  is  running  in  problem 
state.  Therefore*  whenever  it  attempts  a  orivileged 
operation  (SIO*  LPSW,  SSK*  etc.)*  a  program  interrupt 
occurs.  At  this  point*  the  supervisor  (UMMPS)  swaps 
segments  C  and  3  tack  again  and  reconnects  the  other 
segments  ana  transfers  into  the  monitor.  At  this  point 
the  monitor  can  fake  the  privileged  operation  or  not  as 
it  sees  fit. 

The  seauence  is  therefore  like  this*  The  monitor 

transfers  control  to  the  virtual  machine  by  way  of  an 
SVC  SWPTRA.  Any  task  interrupt  in  tne  virtual  machine 
(program  interrupt,  time  overflow,  asynchronous  cevice 
exit,  etc.)  will  cause  the  supervisor  to  re-establish 
standard  addressirg  and  transfer  into  the  monitor.  [60] 


Thus,  SVC  SWPTPA  comes  fairly  close  to  being  a  primitive 
that  starts  up  a  subprocess.  In  its  case,  the  ©articular  choice 
of  which  interrupts  trap  back  to  the  virtual  macnine  supervisor 
nave  been  decided  in  advance.  There  is  no  general  flexibility  in 
deciding  more  precisely  what  is  permittee  in  the  subprocess. 
However,  since  the  subprocess  is  a  particular  kina  of  virtual 
machine,  e.g.  System/360  (360/65),  there  is  no  real  need  to 
provide  the  additional  flexibility. 

Note  that  the  UMMPS  virtual  machine  is  not 
self-virtualizing.  That  is,  the  VMM  runs  on  a  360/67  to  produce 
a  virtual  360/65.  The  work  is  currently  being  extended  to 
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include  a  virtual  360/67. 


C.3  OS/360 


OS/360  166*84]  is  the  principal  operating  system  Deirg  run 
with  the  IBP  System/360.  As  such  it  is  extremely  difficult  (and 
possibly  foolish)  to  implement  ad  hoc  charges  in  it.  One  of  the 
principal  reasons  that  CS/360  is  run  is  so  that  3n  instal  lation 
will  be  somewhat  compatible  with  the  rest  of  the  IBM  world. 
Making  ad  hoc  changes  to  OS/360  sabotages  that  objective. 

Thus*  in  contrast  to  UMMPS*  say*  where  the  operating  system 
was  modified  to  support  virtual  machines*  one  would  not  expect  to 
find  changes  to  OS/360.  This,  in  part  accounts  for  the  lack  of 
Type  II  virtual  machines  under  OS/360  sine"  a  standard  360  is 


Type  I  virtual izab le.  (See  Appendix  8.3 

In  this  section,  we  will  examine  some  aspects  of  CS/360  and 


show  how  the  public  version  of  OS  (pre-1970)  violates  Rules  S2 
and  S3.  He  use  as  a  guide*  "Notes  on  Construction  of  Subsystems 
Hithin  Operating  System/360"  by  E.  Satterthwaite  [100]. 
Satter thwai te* s  objective  is  slightly  different  from  implementing 
virtual  machines*  but  the  difficulties  he  uncovers  are  ecually 
applicable.  Since  OS/360  is  continually  evolving*  remarks  made 
about  it  often  tend  to  be  time-dependent. 

OS/360  violates  rule  S2  because  all  memory  available  to  a 
Job  is  assigned  a  single  key*  and  that  key  appears  in  the  PSW  of 
every  task  associated  with  that  Job.  This  implies  that  the 
virtual  machine  is  able  to  destroy  the  VMM,  Satterthwaite 
suggests  that  OS/360  provide  a  pair  of  SVC  instructions*  which. 
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after  appropriate  validity  checking*  switch  the  protection  key  of 
a  part  of  a  partition  (region)  between  zero  and  the  key 
associated  with  the  corresponding  lob. 

OS/360  violates  rule  S3  in  many  different-  ways.  Of  the 
interrupts  and  exceptions*  only  program  interrupts  may  be 
signalled  to  the  virtual  machine  monitor  in  such  a  way  as  to  be 
recoverable.  For  program  interrupts  in  the  virtual  machine*  e.g. 
attempts  to  execute  privileged  instructions*  the  VMM  is  able  to 
issue  an  SVC  SPIE  to  give  the  address  of  the  aooropriate  “error 
handling**  routine. 

Other  interrupts  such  as  I/O*  timing  or  SVC  interruptions 
are  not  handled  as  well.  For  example*  although  the  virtual 
machine  supervisor  can  specify  a  time  interval  and  an  exit 
routine  to  be  given  control  upon  the  expiration  of  that  interval, 
there  does  not  appear  to  be  a  method  by  which  the  exit  routine 
may  examine  or  alter  the  PSW  or  general  register  contents 
existing  at  the  point  ot  interruption.  Thus,  it  is  impossible  to 
simulate  the  effect  of  this  instruction.  The  worst  failing  of 
0S/36G  concerns  the  lack  of  an  hierarchical  viewpoir.*  concerning 
SVC’s.  There  does  not  appear  to  be  a  way  in  CS  for  a  virtual 
machine  supervisor  to  mask  out  the  SVC’s  of  its  virtual  machire. 
Such  a  mask  should  enable  a  specified  exit  routine  to 
conditionally  inhibit  the  execution  of  selected  supervisor 
services.  Without  this  trap*  there  does  not  aopear  to  be  a  way 
for  the  virtual  machine’s  SVC’s  to  be  signalled*  s  violation  of 
Rule  S3a.  Thus*  in  Figure  C-i*  execution  of  an  SVC  by  the 
virtual  machine  (in  problem  state)  causes  a  hareware  trap  to  the 


?K£««&&^gjj|8$»39Sg§?  s^vr^jgSs^s  VJ’ 


fc^-fT:-  -f.-M  .*^n:c. 


Page  214 

system  and  control  passes  to  0S/36Q.  CS/360  then  interprets  the 
SVC  as  a  request  for  its.  service*  performs  the  service  3nd 
returns  to  the  calling  virtual  machine  (in  oroclem  state)*  Thus 
the  VPk  never  gets  control  and  it  cannot  put  the  virtual  machine 
into  virtual  supervisor  state  for  the  intended  SVC  handing. 
Vole  the  similarity  between  the  neec  to  mask  out  SVC's  of  the 
extended  machine  and  the  need  to  cause  tracs  on  sensitive 
instructions  in  the  hardware  machine.  A  masking  SVC  is,  thus, 
very  similar  to  tne  TOO  instruction  of  Section  3.4. 


C  « 4  MIT  POP-iC  ITS 

While  the  POP-10  is  not  v irtua I izab I e ,  it  is,  however, 
oossible  to  construct  an  HVM  on  it.  {See  Section  3.3  and 
Appendix  8.]  Furthermore,  wher  running  uncer  MIT’s  ITS, 
Incompatible  Timesharing  System  142,43],  it  becomes  feasible  to 
construct  a  Type  II  Hybrid  Virtual  Machine  (HVM)  tS63. 

ITS  is  an  hierarchical  ooerating  system  in  which  one  process 
can  control  a  sub-tree  of  other  crocesses.  In  particular,  ITS 
provides  primitives  (UUC’s)  which  enable  a  process  to  create  a 
subprocess,  read  the  core  image  of  the  subcrocess,  and  get 
aierted  when  the  sufcprocess  attempts  to  reference  outside  its 
core  image  or  attempts  to  execute  a  privileged  instruction  or 
UUC.  The  entira  signalling  mechanism  (user  trao  mode)  is 
controlled  by  one  bit,  called  the  UTPP  bit,  in  the  process’s 
control  block  ("system  variables”)*  A  UUO  to  turn  the  UTRP  bit 

on  or  off  is  available  to  the  parent  process* 

Thus,  it  becomes  a  simole  matter  to  design  a  Tyoe  II  HVM. 
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The  VMM  is  the  superior,  the  virtual  machine  the  interior  in  a 
orocess  tree.  The  VMM  sets  the  UTRP  bit  to  direct  all  inferior 
traos  to  the  VMM.  The  VMM  permits  the  HVM  to  execute  directly  in 
user  mode.  When  the  HVM  attempts  to  enter  virtual  executive 
•node*  the  VMM  is  alerted.  The  VMM  interprets  each  instruction 
until  the  virtual  machine  returns  to  user  mode.  See  Figure  C-2. 

C.5  CHS*  The  Cambridge  Monitor  System 

CMS  { 85 J  is  a  rather  bare  single-user  conversational 
ooeratirg  system  which  runs  or  a  standard  360*  It  is  most  often 
found  in  conjunction  with  CP-67.  In  this  usage*  CMS  runs  on  a 
virtual  machine  under  CP-67.  CMS  provides  a  number  of  operating 
system  primitives  that  may  be  used  by  a  orospective  VMM. 
However*  since  CMS  is  a  single  user  conversational  system  (most 
often  run  on  a  virtual  machine),  it  does  not  provide  any  features 
that  may  be  used  to  protect  or  signal  the  VMM  as  required  by 
rules  S2  and  S3.  At  the  same  time*  CMS  does  not  make  the 
Hardware  features  (instructions)  to  accomplish  tnese  tasks 
inaccessible.  Thus,  with  rather  simple  alterations,  the 
necessary  mechanisms  may  be  added  to  CMS.  khile  no  virtual 
machine  under  CMS  has  been  discussed  in  the  literature,  various 
installations,  such  as  MIT  Lincoln  Laboratory,  have  made  most  of 
the  modifications  that  make  this  possible. 
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GLOSSARY 


The  intent  of  the  glossary  is  to  provide *  in  one  ol.ace*  very 


nri€v  sugary  definitions  fer  sore  of  the  terms  or  abbrevi at i ors 


jsec  ir  the  thesis.  Per  this  reason  there  is  ro  attamsf  fade  to 


live  very  complete  or  oar t icu lar I v  formsj  definitions.  This 


ilosssry  is  merely  intended  ns  a  convenience  for  toe  reader. 


''etinrlirq  him  of  cerfsin  terms  cr  ebbrev  i  a  t  i  ar.s.  For  more 


inf  emotion*  the  reader  is  directed  to  4hs  section  numbers 


indicated  with  most  entries. 


4££Ci.iSSii!r£ ;  attributes  of  a  system  as  sren  hy  The  o^ogr a-mmer* 
i  .e.  the  cor.ceotual  structure  and  functional  behavior*  as 
listirrt  frorr  the  organization  of  the  data  flow  ard  controls*  i  he 
logical  design*  and  the  ohysical  implementation 


assoc  i ator  t  hardware  device  f  cr  performing  rapid  oarallel  search 


pontro  |  program  !  virtual  machine  men  it  or 


£  mw  i  exterdec  machine  monitor  mapping  pro;r3m  betweer  two 

different  operating  systems  t2.11 


emulatjoni  techniaue  for  simulating  an  interior  decor  on  a 
machine  with  some  .different  one 


except i on?  trap  caused  by  process  violation  [different  from 
fault]!  cortrol  should  remain  within  executing  virtual  machine 
[4.11 


iaiilyf  compatible  series  of  computers  wits  possibly  minor 
interior  decor  differencas*  e.q.  36C/40  anc*  3*1/57, 


fgrj.|y*virtual  i  zi  no !  the  virtual  machine  is  nut  identic3l  io  the 
nost  but  is  a  member  of  the  same  computer  family  C2.11 
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iiniIiL2C.fi !  rrj croproprs*”  which  defines  interior  :?cor 
IV!  f  •'  <ri  I  y-virtua  I  isih}  12.11 
f:  vi<"+u.-l  macnine  mao  [4.21 

i  t  croc  s s  man  C4.ll 

J5C^r^ii5G !  architectural  character ist ics  cf  the  orircical 
13 chines  of  tost  generation  tin  the  thesis*  cane ration  always 
reftr^  to  areni  tec  t  u<~-i  *  never  to  physical  imp  I erartst ion,  etc.l 

H V «  horoware  virtual izer  14.5) 

dV£t  hyori.-j  virtual  machine  I .? .  3  5 

larawars  virtual i zer i  collective  title  for  n irdwsra-f irmwere 
design  which  directly  suooo^ts  the  (IV  oe.nerat  ior>)  VCS  mcdel 
C4.r] 

G2Si  machine!  oar*  machine  on  which  virtual  machine  moo i for  runo 

C2.ll 

222 1  ooorati n  j  system:  cnerstinc  system  under  which  Tyoe  II 

virtual  machine  monitor  rurs  C2.ll 

nyfiEi.o  virtual  virtual  machine  in  which  all 

instructions  issued  within  the  most  privilegeo  lever  are  software 
inierorated  an*  all  other  instructions  execute  directly  [3.31 

IX  generation!  seconc  generation  architectures  tyoicef  cf  I  PM 
r  190  [4. »] 

III  oer  erat ion!  third  generation  architectures  tvoical  of  IBM 
SM  C3.1: 

LI  asnigratis.Q :  fourth  generation  archi  t  ect  ur  as  with  elaborate, 

-ich  nrccess  structure  imolemerted  in  firmware  [4.11 

XolSCifiC  fj^cqri  sof  tware-visiM  e  definition  of  3  system,  i.e. 
3rcri tectur& 

i.2 Ir^zXfySl  aafi!  mao  wifhir.  s  virtual  machine  level  which 
establishes  the  (layer)  structure  within  that  level!  the  mac  may 
oe  visible  1c  privileged  software  of  that  level  C4.1,4.21 

i ntei —  level  mao:  (VMmao)  map  between  two  levels  of  virtual 

machines,  e.c.  level  r,  to  level  n  +  1*  which  is  invisible  tc  the 
higher  level,  e.q.  level  n+1  [4.21 

invjsjg  K>:  unable  to  be  detected  ir  software 
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i3Y£CJ  restricted  accjss  structure  (within  3  level)  such  as 
iaster/slave  modes  or  rings  [2.1,4.?] 


l2^el  i  number  of  Vhmaos  which  must  be  successively  aoclied  to 
nap  ;•  virtual  resource  name  into  a  real  resource  ns-net  also 
corresponds  to  the  number  of  syllables  in  the  i^IOt  thus,  the 
I  mc-chin*  is  level  C 


gat jye  irpdet  operation  of  system,  e.g.  CPU,  with  preferred  order 
code,  rot  emulation 


in ce£  £c o e  :  instruction  set 


2£t?t  process  control  oIock,  orocess  status  kept  in  system  base 

t4.ll 


3-n;,inp;  name  used  cv  process  to  refer  to  some  entity 


process  maa*  abstraction  of  the  map  from  process  names,  e.g. 
segment  numters,  to  resource  names,  e.g.  remory  locations  [4.1] 


grjyj  If  r,ed  ins  true  t  i  cn  i  instruction  which  exacutes  (correctly) 
only  in  privileged  mode,  i.e.  supervisor  state,  ''ino  ?ero  [3.11 


7 -name i  real  resource  rame  [4.?] 


relocation-pound  form  of  address  relocation,  as  ir  CEC 

3DF-ir 


ge  I  cadan  le  contro  I  frt  gre :  modifiable  micrerr  oc  =  ssor  storage 
n escurc e  m 5£t  virtual  machine  map 


dies*  layered  protection  system,  generalization  of  m as  ter /s I ave 
mode,  suoervi sor/oroo I em  state  [4.1] 


"inp  1*  most  privileged  ring  [4.1] 


?CjQIt  oriqin  pointer  of  system  base  of  typical  IV  generation 
machine  [4.1] 


running  process  word,  in  system  base,  contains  identifier 
of  active  process  [4.6] 


self-virtualizing  [2.11 


sel  f-virtualizings  the  processor  of  the  virtual  coirrv.ter  system 
is  identical  to  the  host  (2.11 


semaphore*  means  for  interprocess  communication  in  typical  TV 
generation  computer  system  [4.1, 4.61 
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-i££§lii.¥§  IDS  t  xsJq  t  instruction  which,  :»c30Si  of  711 
1  \»r°r  =  1 1  c»-  virtu?  I  machine  software  c  cns  true  t  i  on  will  give 
incorrect  '*>-sul  1  if  oermitted  to  execute  orderly  on  the  ►'cst 
nachine  [  3  .  £ ) 

2U£££vi.i2£  £2iiJ  (gVC)  instruction  which  is  usrat  to  cum-ruriccte 
.iitn  tre  operating  'ystei  f.*.i) 

system  base:  cia+ab-se  oirectly  accessed  by  tinware  ( 3 n  1  updated 
ov  privilege  software)  of  typical  TV'  generation  system  to 
suoper  t  i  ran  leaer.t  at  i  or.  of  the  process  morel  t4.1) 

IOC :  tr  in  cr.  occodo  instruction,  an  hoc  suggestion  for  assigning 

a  (ITT  nenerot i on)  .Thcnire  in  which  orivileceg  instructions  are 
chosen  ft  execution  *ime,  and  thus  can  cuar~ntjs  III  jen^ration 
software  virtual  machine  construction  [3.4] 

l¥££  i  V^MS  the  V MK  rur*  cn  ?  DC-re  machine  hnsf  (?.!] 

IY££  II  VVK:  the  \Jvy  runs  on  en  extencec!  macnire  host,  under  the 
nost  oocTotirg  system 

UHO:  supervisor  call  instruction  on  EEC  P£P-10 

VCS :  virtual  con  outer  system 

VCS  control  blech  in  system  base  of  hardware  virtual  izer 
irovi^cs  Wmcn  for  virtual  machine  th.Cl 

!l£Sl£5:  VCS  taolt  ir  system  base  of  hardware  virtual i?er 

contains  coirters  to  VC$C)*s  for  virtuf  ’  m.-cninss  [4.;] 

±Hi  victual  m chine  in over  virtu;  I  memory) 

VM£R:  S' me  s  VC  SC  IT  T4.3] 

i£I£s  virtual  nscnii?  icentifier  register  in  nardware 
virtu  li-'cr  ircicatcs  active  virtual  machine  [4.~  1 

Inao  Vhin  instruction  in  hardware  virtual i?sr*  apoenos 
next  syllable  to  the  VhlO  register  and  dispatches  tne  virtual 
nachine  [h,sj 

VM* :  virtual  rr  ..chine  vonitor,  software  which  mediates  between 
the  virtual  machine  and  host  12,1) 

virtual  machine  timesharing  machine  [2.c] 

VP:  Verice  ;>aoer  cr  orcoosci:  proposal  for  firmware  assisted 

virtualizer  made  in  151)  14.4) 

VzDSffiei  virtual  resource  heme  14.2) 
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